2024-09-17T05:17:32.333 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T05:17:32.357 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T05:17:32.565 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618 branch: wip-pdonnell-testing-20240916.200549-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7908618' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240916.200549-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: client: client_shutdown_timeout: 120 global: ms die on skipped message: false mds: mds oft prefetch dirfrags: true mds_sleep_rank_change: 5000000.0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Replacing daemon mds - SLOW_OPS - slow request sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240916.200549-debug sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 476 sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf sleep_before_teardown: 0 subset: 320/512 suite: fs suite_branch: wip-pdonnell-testing-20240916.200549-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOtGJUVP3tsnt+XrOJimpa8Dq+T2MMavQtRhjpphZ5mfNej7vtLfRjhCIT+/trd4eM1sedNe258MdPbDAkAQMY= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpQUKABIPRs1JUblvm15W4ZuknnO0I6pA32xlSAfhqx6vTWx1NN9Lq6vx1OOa5ktYYLpx3w4HDr3XbSGsgw2fg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_02:09:51 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T05:17:32.565 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa; will attempt to use it 2024-09-17T05:17:32.566 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks 2024-09-17T05:17:32.566 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T05:17:32.568 INFO:teuthology.task.internal:Checking packages... 2024-09-17T05:17:32.587 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf' 2024-09-17T05:17:32.587 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T05:17:32.587 INFO:teuthology.packaging:ref: None 2024-09-17T05:17:32.587 INFO:teuthology.packaging:tag: None 2024-09-17T05:17:32.587 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240916.200549-debug 2024-09-17T05:17:32.587 INFO:teuthology.packaging:sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:17:32.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240916.200549-debug 2024-09-17T05:17:32.722 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4989.gce1b342b 2024-09-17T05:17:32.724 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T05:17:32.737 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T05:17:32.737 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T05:17:32.751 INFO:teuthology.task.internal:Saving configuration 2024-09-17T05:17:32.769 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T05:17:32.776 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T05:17:32.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 05:12:34.387525', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOtGJUVP3tsnt+XrOJimpa8Dq+T2MMavQtRhjpphZ5mfNej7vtLfRjhCIT+/trd4eM1sedNe258MdPbDAkAQMY='} 2024-09-17T05:17:32.823 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 05:12:34.389855', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpQUKABIPRs1JUblvm15W4ZuknnO0I6pA32xlSAfhqx6vTWx1NN9Lq6vx1OOa5ktYYLpx3w4HDr3XbSGsgw2fg='} 2024-09-17T05:17:32.823 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T05:17:32.830 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-17T05:17:32.830 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-09-17T05:17:32.830 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T05:17:32.923 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8e20bfbac0>, signals=[15]) 2024-09-17T05:17:32.923 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T05:17:32.938 INFO:teuthology.task.internal:Opening connections... 2024-09-17T05:17:32.939 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:17:32.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:17:33.016 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:17:33.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:17:33.089 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T05:17:33.096 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-09-17T05:17:33.114 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-09-17T05:17:33.115 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-09-17T05:17:33.172 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-09-17T05:17:33.172 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-09-17T05:17:33.172 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T05:17:33.173 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T05:17:33.174 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-09-17T05:17:33.198 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-09-17T05:17:33.217 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-09-17T05:17:33.217 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-09-17T05:17:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T05:17:33.276 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-09-17T05:17:33.276 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T05:17:33.276 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T05:17:33.276 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T05:17:33.276 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-09-17T05:17:33.306 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T05:17:33.316 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T05:17:33.348 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T05:17:33.348 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-09-17T05:17:33.350 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-09-17T05:17:33.368 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T05:17:33.395 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T05:17:33.395 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-09-17T05:17:33.408 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-09-17T05:17:33.451 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T05:17:33.507 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T05:17:33.568 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-17T05:17:33.568 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-17T05:17:33.568 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-17T05:17:33.568 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T05:17:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T05:17:33.586 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-09-17T05:17:33.644 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-503.el9.x86_64 2024-09-17T05:17:33.645 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-503.el9.x86_64 2024-09-17T05:17:33.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-09-17T05:17:37.037 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-17T05:17:40.141 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:02 2024-09-17T05:17:46.229 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.5 MB 00:02 2024-09-17T05:17:47.887 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-17T05:17:48.958 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-17T05:17:54.355 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T05:17:56.262 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T05:17:56.262 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T05:17:56.353 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T05:17:56.365 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 79 M 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 126 M 2024-09-17T05:17:56.366 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-17T05:17:56.490 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 461 kB/s | 36 kB 00:00 2024-09-17T05:17:57.182 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-17T05:17:57.673 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-17T05:17:58.607 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-17T05:17:58.607 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-17T05:17:58.607 INFO:teuthology.orchestra.run.smithi117.stdout:Total 35 MB/s | 79 MB 00:02 2024-09-17T05:17:59.291 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-17T05:17:59.668 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-17T05:17:59.668 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-17T05:18:00.621 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-17T05:18:00.621 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-17T05:18:03.043 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-17T05:18:03.718 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T05:18:03.792 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T05:18:06.542 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T05:18:06.668 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T05:18:11.506 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T05:18:11.543 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T05:18:16.383 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T05:18:54.530 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T05:18:54.568 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T05:18:54.901 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T05:18:54.901 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T05:18:54.901 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T05:18:54.902 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:18:55.748 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-17T05:18:56.087 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T05:18:56.175 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T05:18:56.175 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T05:18:56.175 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T05:18:56.175 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T05:18:56.176 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T05:18:56.176 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-17T05:18:56.176 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T05:18:56.176 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T05:18:56.235 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T05:18:56.236 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T05:18:56.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-09-17T05:18:56.272 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-09-17T05:18:56.272 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 82.el9 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8274670 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T05:18:56.273 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:18:56.274 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T05:18:56.275 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T05:18:56.275 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-508.el9.x86_64 2024-09-17T05:18:56.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T05:18:56.939 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-09-17T05:18:58.264 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T05:18:58.304 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-09-17T05:18:58.306 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-09-17T05:18:58.322 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.aYSIiEjiny 2024-09-17T05:18:58.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aYSIiEjiny 2024-09-17T05:18:58.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.aYSIiEjiny 2024-09-17T05:18:58.533 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.aYSIiEjiny is 6KB 2024-09-17T05:18:58.582 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.aYSIiEjiny 2024-09-17T05:18:58.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T05:18:58.659 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T05:18:58.660 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T05:18:58.660 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T05:18:58.660 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T05:18:58.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T05:18:58.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-09-17T05:19:28.816 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-17T05:19:28.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:19:28.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:19:47.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-09-17T05:19:56.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:19:56.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:19:59.325 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-09-17T05:20:11.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:20:11.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:20:29.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-09-17T05:20:44.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:20:44.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:20:45.134 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-09-17T05:20:45.655 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:20:45.655 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-508.el9.x86_64"... 2024-09-17T05:20:45.655 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-09-17T05:20:45.670 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-508.el9.x86_64 2024-09-17T05:20:45.671 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T05:20:45.671 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T05:20:45.671 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T05:20:46.672 DEBUG:teuthology.parallel:result is None 2024-09-17T05:20:46.672 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T05:20:46.679 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T05:20:46.680 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T05:20:46.682 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T05:20:46.702 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T05:20:46.708 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T05:20:46.734 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T05:20:46.735 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T05:20:46.737 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T05:20:46.775 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T05:20:46.781 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T05:20:46.782 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T05:20:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T05:20:46.827 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T05:20:46.829 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T05:20:46.916 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T05:20:46.936 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T05:20:46.940 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T05:20:46.981 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T05:20:46.981 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T05:20:46.988 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T05:20:46.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T05:20:46.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T05:20:47.017 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T05:20:47.044 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T05:20:47.045 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T05:20:47.055 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T05:20:47.073 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T05:20:47.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T05:20:47.212 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T05:20:47.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T05:20:47.294 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:20:47.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T05:20:47.358 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T05:20:47.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T05:20:47.453 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T05:20:47.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T05:20:47.578 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:20:47.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T05:20:47.643 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-09-17T05:20:47.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-09-17T05:20:47.695 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T05:20:47.721 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T05:20:48.198 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T05:20:48.207 INFO:teuthology.task.internal:Starting timer... 2024-09-17T05:20:48.207 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T05:20:48.274 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T05:20:48.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-09-17T05:20:48.390 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-09-17T05:20:48.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-09-17T05:20:48.433 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-09-17T05:20:48.435 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T05:20:48.435 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-09-17T05:20:48.456 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-09-17T05:20:48.456 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-09-17T05:20:48.501 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-09-17T05:20:48.501 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive'} 2024-09-17T05:20:48.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T05:20:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T05:20:48.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T05:20:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T05:20:48.568 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T05:20:48.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-09-17T05:20:48.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-09-17T05:20:48.648 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T05:20:48.658 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T05:20:48.669 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T05:20:48.670 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com,smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T05:24:42.103 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2024-09-17T05:24:42.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:24:42.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:24:42.191 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-09-17T05:24:42.276 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-17T05:24:42.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-17T05:24:42.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T05:24:42.355 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-17T05:24:42.433 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-17T05:24:42.433 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T05:24:42.446 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T05:24:42.446 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T05:24:42.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T05:24:42.451 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T05:24:42.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T05:24:42.479 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T05:24:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T05:24:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T05:24:42.518 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-09-17T05:24:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T05:24:42.528 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-09-17T05:24:42.543 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T05:24:42.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-09-17T05:24:42.556 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-09-17T05:24:42.556 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T05:24:42.570 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T05:24:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T05:24:42.615 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-09-17T05:24:42.619 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T05:24:42.619 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-09-17T05:24:42.619 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.619 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.619 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.619 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-17T05:24:42.643 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T05:24:42.644 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-17T05:24:42.644 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.644 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.644 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.644 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T05:24:42.645 INFO:teuthology.run_tasks:Running task install... 2024-09-17T05:24:42.656 DEBUG:teuthology.task.install:project ceph 2024-09-17T05:24:42.656 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}} 2024-09-17T05:24:42.656 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'} 2024-09-17T05:24:42.656 INFO:teuthology.task.install:Using flavor: default 2024-09-17T05:24:42.664 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T05:24:42.664 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-17T05:24:42.665 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'tag': None, 'wait_for_package': False} 2024-09-17T05:24:42.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:24:42.670 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'tag': None, 'wait_for_package': False} 2024-09-17T05:24:42.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:24:42.820 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/ 2024-09-17T05:24:42.821 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4989.gce1b342b 2024-09-17T05:24:42.836 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/ 2024-09-17T05:24:42.837 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4989.gce1b342b 2024-09-17T05:24:43.001 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T05:24:43.001 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:24:43.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T05:24:43.027 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-17T05:24:43.027 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T05:24:43.047 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T05:24:43.047 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:24:43.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T05:24:43.072 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-17T05:24:43.073 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T05:24:43.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T05:24:43.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T05:24:43.168 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T05:24:43.213 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T05:24:43.232 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-09-17T05:24:43.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-09-17T05:24:43.277 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-09-17T05:24:43.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-09-17T05:24:43.462 INFO:teuthology.orchestra.run.smithi117.stdout:45 files removed 2024-09-17T05:24:43.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-17T05:24:43.504 INFO:teuthology.orchestra.run.smithi119.stdout:45 files removed 2024-09-17T05:24:43.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-17T05:24:44.043 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 315 kB/s | 84 kB 00:00 2024-09-17T05:24:44.069 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 320 kB/s | 84 kB 00:00 2024-09-17T05:24:44.461 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 64 kB/s | 24 kB 00:00 2024-09-17T05:24:44.469 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 67 kB/s | 24 kB 00:00 2024-09-17T05:24:44.737 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-17T05:24:44.737 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-17T05:24:45.287 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-17T05:24:45.353 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-17T05:24:47.763 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-17T05:24:49.487 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 20 MB 00:03 2024-09-17T05:24:51.947 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-17T05:24:53.572 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-17T05:24:54.602 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-17T05:24:55.696 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.5 MB 00:02 2024-09-17T05:24:57.046 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 142 kB/s | 19 kB 00:00 2024-09-17T05:24:58.050 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-17T05:25:00.024 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T05:25:02.143 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T05:25:02.143 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T05:25:02.155 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-17T05:25:02.156 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-17T05:25:02.156 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-17T05:25:02.157 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-17T05:25:02.159 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-17T05:25:02.161 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-17T05:25:02.162 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-17T05:25:02.226 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-17T05:25:02.246 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-17T05:25:02.246 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-17T05:25:02.246 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-17T05:25:02.246 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-17T05:25:02.246 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 6.6 k 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.5 M 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 873 k 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 148 k 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 1.5 M 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 154 k 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 14 M 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 7.4 M 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 49 k 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 10 M 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 40 M 2024-09-17T05:25:02.247 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 285 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 784 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 225 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 27 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 18 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 32 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 766 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 127 k 2024-09-17T05:25:02.248 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 162 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 321 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 305 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 99 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 86 k 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.1 M 2024-09-17T05:25:02.249 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 170 k 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.4 M 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.2 M 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T05:25:02.250 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 20 M 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 34 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 2.3 M 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 291 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.0 M 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 18 M 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 16 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 25 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T05:25:02.251 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 159 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T05:25:02.252 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 507 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.3 M 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T05:25:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 45 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 143 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T05:25:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T05:25:02.255 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T05:25:02.256 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T05:25:02.257 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T05:25:02.258 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:02.259 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-17T05:25:02.260 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-17T05:25:02.260 INFO:teuthology.orchestra.run.smithi119.stdout:Install 149 Packages 2024-09-17T05:25:02.260 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 8 Packages 2024-09-17T05:25:02.260 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:02.260 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 228 M 2024-09-17T05:25:02.261 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-17T05:25:02.618 INFO:teuthology.orchestra.run.smithi119.stdout:(1/157): ceph-19.3.0-4989.gce1b342b.el9.x86_64. 51 kB/s | 6.6 kB 00:00 2024-09-17T05:25:02.851 INFO:teuthology.orchestra.run.smithi119.stdout:(2/157): ceph-fuse-19.3.0-4989.gce1b342b.el9.x8 3.7 MB/s | 873 kB 00:00 2024-09-17T05:25:02.968 INFO:teuthology.orchestra.run.smithi119.stdout:(3/157): ceph-base-19.3.0-4989.gce1b342b.el9.x8 12 MB/s | 5.5 MB 00:00 2024-09-17T05:25:02.993 INFO:teuthology.orchestra.run.smithi119.stdout:(4/157): ceph-immutable-object-cache-19.3.0-498 1.0 MB/s | 148 kB 00:00 2024-09-17T05:25:03.077 INFO:teuthology.orchestra.run.smithi119.stdout:(5/157): ceph-mds-19.3.0-4989.gce1b342b.el9.x86 21 MB/s | 2.3 MB 00:00 2024-09-17T05:25:03.152 INFO:teuthology.orchestra.run.smithi119.stdout:(6/157): ceph-mgr-19.3.0-4989.gce1b342b.el9.x86 9.6 MB/s | 1.5 MB 00:00 2024-09-17T05:25:03.369 INFO:teuthology.orchestra.run.smithi119.stdout:(7/157): ceph-mon-19.3.0-4989.gce1b342b.el9.x86 17 MB/s | 5.0 MB 00:00 2024-09-17T05:25:03.405 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T05:25:03.661 INFO:teuthology.orchestra.run.smithi119.stdout:(8/157): ceph-common-19.3.0-4989.gce1b342b.el9. 17 MB/s | 20 MB 00:01 2024-09-17T05:25:03.694 INFO:teuthology.orchestra.run.smithi119.stdout:(9/157): ceph-selinux-19.3.0-4989.gce1b342b.el9 760 kB/s | 25 kB 00:00 2024-09-17T05:25:03.920 INFO:teuthology.orchestra.run.smithi119.stdout:(10/157): ceph-radosgw-19.3.0-4989.gce1b342b.el 19 MB/s | 10 MB 00:00 2024-09-17T05:25:03.970 INFO:teuthology.orchestra.run.smithi119.stdout:(11/157): cephfs-mirror-19.3.0-4989.gce1b342b.e 4.4 MB/s | 225 kB 00:00 2024-09-17T05:25:04.028 INFO:teuthology.orchestra.run.smithi119.stdout:(12/157): cephfs-shell-19.3.0-4989.gce1b342b.el 463 kB/s | 27 kB 00:00 2024-09-17T05:25:04.079 INFO:teuthology.orchestra.run.smithi119.stdout:(13/157): libcephfs-devel-19.3.0-4989.gce1b342b 638 kB/s | 32 kB 00:00 2024-09-17T05:25:04.154 INFO:teuthology.orchestra.run.smithi119.stdout:(14/157): libcephfs2-19.3.0-4989.gce1b342b.el9. 10 MB/s | 766 kB 00:00 2024-09-17T05:25:04.221 INFO:teuthology.orchestra.run.smithi119.stdout:(15/157): libcephsqlite-19.3.0-4989.gce1b342b.e 2.3 MB/s | 159 kB 00:00 2024-09-17T05:25:04.288 INFO:teuthology.orchestra.run.smithi119.stdout:(16/157): librados-devel-19.3.0-4989.gce1b342b. 1.9 MB/s | 127 kB 00:00 2024-09-17T05:25:04.371 INFO:teuthology.orchestra.run.smithi119.stdout:(17/157): libradosstriper1-19.3.0-4989.gce1b342 6.0 MB/s | 507 kB 00:00 2024-09-17T05:25:04.655 INFO:teuthology.orchestra.run.smithi119.stdout:(18/157): librgw2-19.3.0-4989.gce1b342b.el9.x86 19 MB/s | 5.3 MB 00:00 2024-09-17T05:25:04.856 INFO:teuthology.orchestra.run.smithi119.stdout:(19/157): ceph-osd-19.3.0-4989.gce1b342b.el9.x8 10 MB/s | 18 MB 00:01 2024-09-17T05:25:04.881 INFO:teuthology.orchestra.run.smithi119.stdout:(20/157): python3-ceph-argparse-19.3.0-4989.gce 200 kB/s | 45 kB 00:00 2024-09-17T05:25:04.914 INFO:teuthology.orchestra.run.smithi119.stdout:(21/157): python3-ceph-common-19.3.0-4989.gce1b 2.4 MB/s | 143 kB 00:00 2024-09-17T05:25:04.939 INFO:teuthology.orchestra.run.smithi119.stdout:(22/157): python3-cephfs-19.3.0-4989.gce1b342b. 2.7 MB/s | 162 kB 00:00 2024-09-17T05:25:04.973 INFO:teuthology.orchestra.run.smithi119.stdout:(23/157): python3-rados-19.3.0-4989.gce1b342b.e 5.4 MB/s | 321 kB 00:00 2024-09-17T05:25:05.007 INFO:teuthology.orchestra.run.smithi119.stdout:(24/157): python3-rbd-19.3.0-4989.gce1b342b.el9 4.4 MB/s | 305 kB 00:00 2024-09-17T05:25:05.116 INFO:teuthology.orchestra.run.smithi119.stdout:(25/157): python3-rgw-19.3.0-4989.gce1b342b.el9 695 kB/s | 99 kB 00:00 2024-09-17T05:25:05.183 INFO:teuthology.orchestra.run.smithi119.stdout:(26/157): rbd-fuse-19.3.0-4989.gce1b342b.el9.x8 492 kB/s | 86 kB 00:00 2024-09-17T05:25:05.233 INFO:teuthology.orchestra.run.smithi119.stdout:(27/157): rbd-nbd-19.3.0-4989.gce1b342b.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-17T05:25:05.300 INFO:teuthology.orchestra.run.smithi119.stdout:(28/157): ceph-grafana-dashboards-19.3.0-4989.g 515 kB/s | 34 kB 00:00 2024-09-17T05:25:05.376 INFO:teuthology.orchestra.run.smithi119.stdout:(29/157): rbd-mirror-19.3.0-4989.gce1b342b.el9. 12 MB/s | 3.1 MB 00:00 2024-09-17T05:25:05.409 INFO:teuthology.orchestra.run.smithi119.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4989.gce1b342 1.4 MB/s | 154 kB 00:00 2024-09-17T05:25:05.568 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T05:25:05.569 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T05:25:05.583 INFO:teuthology.orchestra.run.smithi117.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-17T05:25:05.584 INFO:teuthology.orchestra.run.smithi117.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-17T05:25:05.585 INFO:teuthology.orchestra.run.smithi117.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-17T05:25:05.586 INFO:teuthology.orchestra.run.smithi117.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-17T05:25:05.588 INFO:teuthology.orchestra.run.smithi117.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-17T05:25:05.591 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-17T05:25:05.592 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-17T05:25:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 6.6 k 2024-09-17T05:25:05.666 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.5 M 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 873 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 148 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 1.5 M 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 154 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 14 M 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 7.4 M 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 49 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 10 M 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 40 M 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 285 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 784 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 225 k 2024-09-17T05:25:05.667 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 27 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 18 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 32 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 766 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 127 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-17T05:25:05.668 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 162 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 321 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 305 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 99 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 86 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.1 M 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 170 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-09-17T05:25:05.669 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.4 M 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.2 M 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T05:25:05.670 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 20 M 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 34 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 2.3 M 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 291 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.0 M 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 18 M 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 16 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 25 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T05:25:05.671 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 159 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 507 k 2024-09-17T05:25:05.672 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.3 M 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T05:25:05.673 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 45 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 143 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T05:25:05.674 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T05:25:05.675 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T05:25:05.676 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T05:25:05.677 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T05:25:05.678 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:05.679 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-17T05:25:05.680 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-09-17T05:25:05.680 INFO:teuthology.orchestra.run.smithi117.stdout:Install 149 Packages 2024-09-17T05:25:05.680 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 8 Packages 2024-09-17T05:25:05.680 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:05.680 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 228 M 2024-09-17T05:25:05.680 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-17T05:25:05.776 INFO:teuthology.orchestra.run.smithi119.stdout:(31/157): ceph-test-19.3.0-4989.gce1b342b.el9.x 19 MB/s | 40 MB 00:02 2024-09-17T05:25:05.835 INFO:teuthology.orchestra.run.smithi119.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4989.gce 4.9 MB/s | 291 kB 00:00 2024-09-17T05:25:05.893 INFO:teuthology.orchestra.run.smithi119.stdout:(33/157): ceph-mgr-rook-19.3.0-4989.gce1b342b.e 849 kB/s | 49 kB 00:00 2024-09-17T05:25:05.994 INFO:teuthology.orchestra.run.smithi119.stdout:(34/157): ceph-mgr-diskprediction-local-19.3.0- 13 MB/s | 7.4 MB 00:00 2024-09-17T05:25:06.019 INFO:teuthology.orchestra.run.smithi119.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4989.gc 132 kB/s | 16 kB 00:00 2024-09-17T05:25:06.044 INFO:teuthology.orchestra.run.smithi119.stdout:(36/157): ceph-volume-19.3.0-4989.gce1b342b.el9 5.6 MB/s | 285 kB 00:00 2024-09-17T05:25:06.094 INFO:teuthology.orchestra.run.smithi119.stdout:(37/157): cephadm-19.3.0-4989.gce1b342b.el9.noa 10 MB/s | 784 kB 00:00 2024-09-17T05:25:06.120 INFO:teuthology.orchestra.run.smithi119.stdout:(38/157): cephfs-top-19.3.0-4989.gce1b342b.el9. 243 kB/s | 18 kB 00:00 2024-09-17T05:25:06.262 INFO:teuthology.orchestra.run.smithi119.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 478 kB/s | 80 kB 00:00 2024-09-17T05:25:06.404 INFO:teuthology.orchestra.run.smithi119.stdout:(40/157): ceph-mgr-dashboard-19.3.0-4989.gce1b3 14 MB/s | 14 MB 00:01 2024-09-17T05:25:06.429 INFO:teuthology.orchestra.run.smithi119.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-17T05:25:06.523 INFO:teuthology.orchestra.run.smithi117.stdout:(1/157): ceph-19.3.0-4989.gce1b342b.el9.x86_64. 50 kB/s | 6.6 kB 00:00 2024-09-17T05:25:06.529 INFO:teuthology.orchestra.run.smithi119.stdout:(42/157): libgfortran-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 801 kB 00:00 2024-09-17T05:25:06.588 INFO:teuthology.orchestra.run.smithi119.stdout:(43/157): python3-cryptography-36.0.1-4.el9.x86 7.7 MB/s | 1.2 MB 00:00 2024-09-17T05:25:06.613 INFO:teuthology.orchestra.run.smithi119.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-17T05:25:06.663 INFO:teuthology.orchestra.run.smithi119.stdout:(45/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 978 kB/s | 253 kB 00:00 2024-09-17T05:25:06.688 INFO:teuthology.orchestra.run.smithi119.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-09-17T05:25:06.714 INFO:teuthology.orchestra.run.smithi119.stdout:(47/157): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-17T05:25:06.747 INFO:teuthology.orchestra.run.smithi119.stdout:(48/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-17T05:25:06.749 INFO:teuthology.orchestra.run.smithi117.stdout:(2/157): ceph-fuse-19.3.0-4989.gce1b342b.el9.x8 3.8 MB/s | 873 kB 00:00 2024-09-17T05:25:06.865 INFO:teuthology.orchestra.run.smithi117.stdout:(3/157): ceph-base-19.3.0-4989.gce1b342b.el9.x8 12 MB/s | 5.5 MB 00:00 2024-09-17T05:25:06.890 INFO:teuthology.orchestra.run.smithi117.stdout:(4/157): ceph-immutable-object-cache-19.3.0-498 1.0 MB/s | 148 kB 00:00 2024-09-17T05:25:06.975 INFO:teuthology.orchestra.run.smithi117.stdout:(5/157): ceph-mds-19.3.0-4989.gce1b342b.el9.x86 21 MB/s | 2.3 MB 00:00 2024-09-17T05:25:07.050 INFO:teuthology.orchestra.run.smithi117.stdout:(6/157): ceph-mgr-19.3.0-4989.gce1b342b.el9.x86 9.6 MB/s | 1.5 MB 00:00 2024-09-17T05:25:07.275 INFO:teuthology.orchestra.run.smithi117.stdout:(7/157): ceph-mon-19.3.0-4989.gce1b342b.el9.x86 17 MB/s | 5.0 MB 00:00 2024-09-17T05:25:07.289 INFO:teuthology.orchestra.run.smithi119.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 197 kB/s | 107 kB 00:00 2024-09-17T05:25:07.398 INFO:teuthology.orchestra.run.smithi119.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 200 kB/s | 22 kB 00:00 2024-09-17T05:25:07.540 INFO:teuthology.orchestra.run.smithi119.stdout:(51/157): automake-1.16.2-8.el9.noarch.rpm 810 kB/s | 689 kB 00:00 2024-09-17T05:25:07.558 INFO:teuthology.orchestra.run.smithi117.stdout:(8/157): ceph-common-19.3.0-4989.gce1b342b.el9. 17 MB/s | 20 MB 00:01 2024-09-17T05:25:07.573 INFO:teuthology.orchestra.run.smithi119.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-09-17T05:25:07.600 INFO:teuthology.orchestra.run.smithi117.stdout:(9/157): ceph-selinux-19.3.0-4989.gce1b342b.el9 607 kB/s | 25 kB 00:00 2024-09-17T05:25:07.657 INFO:teuthology.orchestra.run.smithi119.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-09-17T05:25:07.774 INFO:teuthology.orchestra.run.smithi119.stdout:(54/157): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 18 kB 00:00 2024-09-17T05:25:07.799 INFO:teuthology.orchestra.run.smithi119.stdout:(55/157): flex-2.6.4-9.el9.x86_64.rpm 781 kB/s | 313 kB 00:00 2024-09-17T05:25:07.884 INFO:teuthology.orchestra.run.smithi117.stdout:(10/157): ceph-radosgw-19.3.0-4989.gce1b342b.el 17 MB/s | 10 MB 00:00 2024-09-17T05:25:07.891 INFO:teuthology.orchestra.run.smithi119.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-09-17T05:25:07.917 INFO:teuthology.orchestra.run.smithi119.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 1.4 MB/s | 171 kB 00:00 2024-09-17T05:25:07.951 INFO:teuthology.orchestra.run.smithi117.stdout:(11/157): cephfs-mirror-19.3.0-4989.gce1b342b.e 3.3 MB/s | 225 kB 00:00 2024-09-17T05:25:08.025 INFO:teuthology.orchestra.run.smithi119.stdout:(58/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.6 MB/s | 3.0 MB 00:00 2024-09-17T05:25:08.050 INFO:teuthology.orchestra.run.smithi119.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2024-09-17T05:25:08.076 INFO:teuthology.orchestra.run.smithi119.stdout:(60/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 870 kB/s | 160 kB 00:00 2024-09-17T05:25:08.143 INFO:teuthology.orchestra.run.smithi119.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-17T05:25:08.159 INFO:teuthology.orchestra.run.smithi117.stdout:(12/157): ceph-osd-19.3.0-4989.gce1b342b.el9.x8 16 MB/s | 18 MB 00:01 2024-09-17T05:25:08.176 INFO:teuthology.orchestra.run.smithi117.stdout:(13/157): cephfs-shell-19.3.0-4989.gce1b342b.el 120 kB/s | 27 kB 00:00 2024-09-17T05:25:08.185 INFO:teuthology.orchestra.run.smithi119.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 225 kB/s | 24 kB 00:00 2024-09-17T05:25:08.193 INFO:teuthology.orchestra.run.smithi117.stdout:(14/157): libcephfs-devel-19.3.0-4989.gce1b342b 956 kB/s | 32 kB 00:00 2024-09-17T05:25:08.243 INFO:teuthology.orchestra.run.smithi117.stdout:(15/157): libcephfs2-19.3.0-4989.gce1b342b.el9. 11 MB/s | 766 kB 00:00 2024-09-17T05:25:08.252 INFO:teuthology.orchestra.run.smithi119.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-09-17T05:25:08.269 INFO:teuthology.orchestra.run.smithi117.stdout:(16/157): libcephsqlite-19.3.0-4989.gce1b342b.e 2.1 MB/s | 159 kB 00:00 2024-09-17T05:25:08.277 INFO:teuthology.orchestra.run.smithi119.stdout:(64/157): libtool-2.4.6-46.el9.x86_64.rpm 2.5 MB/s | 581 kB 00:00 2024-09-17T05:25:08.302 INFO:teuthology.orchestra.run.smithi117.stdout:(17/157): librados-devel-19.3.0-4989.gce1b342b. 2.1 MB/s | 127 kB 00:00 2024-09-17T05:25:08.303 INFO:teuthology.orchestra.run.smithi119.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-17T05:25:08.336 INFO:teuthology.orchestra.run.smithi117.stdout:(18/157): libradosstriper1-19.3.0-4989.gce1b342 7.4 MB/s | 507 kB 00:00 2024-09-17T05:25:08.361 INFO:teuthology.orchestra.run.smithi119.stdout:(66/157): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-09-17T05:25:08.386 INFO:teuthology.orchestra.run.smithi119.stdout:(67/157): openblas-0.3.26-2.el9.x86_64.rpm 354 kB/s | 38 kB 00:00 2024-09-17T05:25:08.421 INFO:teuthology.orchestra.run.smithi117.stdout:(19/157): python3-ceph-argparse-19.3.0-4989.gce 540 kB/s | 45 kB 00:00 2024-09-17T05:25:08.503 INFO:teuthology.orchestra.run.smithi117.stdout:(20/157): python3-ceph-common-19.3.0-4989.gce1b 1.7 MB/s | 143 kB 00:00 2024-09-17T05:25:08.570 INFO:teuthology.orchestra.run.smithi117.stdout:(21/157): librgw2-19.3.0-4989.gce1b342b.el9.x86 20 MB/s | 5.3 MB 00:00 2024-09-17T05:25:08.595 INFO:teuthology.orchestra.run.smithi117.stdout:(22/157): python3-cephfs-19.3.0-4989.gce1b342b. 1.7 MB/s | 162 kB 00:00 2024-09-17T05:25:08.620 INFO:teuthology.orchestra.run.smithi117.stdout:(23/157): python3-rados-19.3.0-4989.gce1b342b.e 6.3 MB/s | 321 kB 00:00 2024-09-17T05:25:08.646 INFO:teuthology.orchestra.run.smithi117.stdout:(24/157): python3-rbd-19.3.0-4989.gce1b342b.el9 5.9 MB/s | 305 kB 00:00 2024-09-17T05:25:08.671 INFO:teuthology.orchestra.run.smithi117.stdout:(25/157): python3-rgw-19.3.0-4989.gce1b342b.el9 1.9 MB/s | 99 kB 00:00 2024-09-17T05:25:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:(26/157): rbd-fuse-19.3.0-4989.gce1b342b.el9.x8 2.0 MB/s | 86 kB 00:00 2024-09-17T05:25:08.695 INFO:teuthology.orchestra.run.smithi119.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-09-17T05:25:08.739 INFO:teuthology.orchestra.run.smithi117.stdout:(27/157): rbd-nbd-19.3.0-4989.gce1b342b.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-17T05:25:08.822 INFO:teuthology.orchestra.run.smithi117.stdout:(28/157): rbd-mirror-19.3.0-4989.gce1b342b.el9. 21 MB/s | 3.1 MB 00:00 2024-09-17T05:25:08.845 INFO:teuthology.orchestra.run.smithi119.stdout:(69/157): openblas-openmp-0.3.26-2.el9.x86_64.r 9.1 MB/s | 4.9 MB 00:00 2024-09-17T05:25:08.847 INFO:teuthology.orchestra.run.smithi117.stdout:(29/157): ceph-grafana-dashboards-19.3.0-4989.g 316 kB/s | 34 kB 00:00 2024-09-17T05:25:08.873 INFO:teuthology.orchestra.run.smithi117.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4989.gce1b342 3.0 MB/s | 154 kB 00:00 2024-09-17T05:25:08.878 INFO:teuthology.orchestra.run.smithi119.stdout:(70/157): postgresql-contrib-13.16-1.el9.x86_64 4.8 MB/s | 910 kB 00:00 2024-09-17T05:25:09.129 INFO:teuthology.orchestra.run.smithi119.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 484 kB/s | 137 kB 00:00 2024-09-17T05:25:09.340 INFO:teuthology.orchestra.run.smithi117.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 16 MB/s | 7.4 MB 00:00 2024-09-17T05:25:09.732 INFO:teuthology.orchestra.run.smithi117.stdout:(32/157): ceph-test-19.3.0-4989.gce1b342b.el9.x 19 MB/s | 40 MB 00:02 2024-09-17T05:25:09.757 INFO:teuthology.orchestra.run.smithi117.stdout:(33/157): ceph-mgr-modules-core-19.3.0-4989.gce 697 kB/s | 291 kB 00:00 2024-09-17T05:25:09.782 INFO:teuthology.orchestra.run.smithi117.stdout:(34/157): ceph-mgr-rook-19.3.0-4989.gce1b342b.e 992 kB/s | 49 kB 00:00 2024-09-17T05:25:09.808 INFO:teuthology.orchestra.run.smithi117.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4989.gc 329 kB/s | 16 kB 00:00 2024-09-17T05:25:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:(36/157): ceph-volume-19.3.0-4989.gce1b342b.el9 5.6 MB/s | 285 kB 00:00 2024-09-17T05:25:09.983 INFO:teuthology.orchestra.run.smithi117.stdout:(37/157): ceph-mgr-dashboard-19.3.0-4989.gce1b3 12 MB/s | 14 MB 00:01 2024-09-17T05:25:10.008 INFO:teuthology.orchestra.run.smithi117.stdout:(38/157): cephfs-top-19.3.0-4989.gce1b342b.el9. 104 kB/s | 18 kB 00:00 2024-09-17T05:25:10.034 INFO:teuthology.orchestra.run.smithi117.stdout:(39/157): cephadm-19.3.0-4989.gce1b342b.el9.noa 3.4 MB/s | 784 kB 00:00 2024-09-17T05:25:10.101 INFO:teuthology.orchestra.run.smithi117.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 684 kB/s | 80 kB 00:00 2024-09-17T05:25:10.159 INFO:teuthology.orchestra.run.smithi117.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-17T05:25:10.184 INFO:teuthology.orchestra.run.smithi117.stdout:(42/157): libgfortran-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 801 kB 00:00 2024-09-17T05:25:10.218 INFO:teuthology.orchestra.run.smithi117.stdout:(43/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-17T05:25:10.235 INFO:teuthology.orchestra.run.smithi117.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-17T05:25:10.260 INFO:teuthology.orchestra.run.smithi117.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-09-17T05:25:10.285 INFO:teuthology.orchestra.run.smithi117.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-17T05:25:10.311 INFO:teuthology.orchestra.run.smithi117.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-09-17T05:25:10.361 INFO:teuthology.orchestra.run.smithi117.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.2 MB 00:00 2024-09-17T05:25:10.495 INFO:teuthology.orchestra.run.smithi117.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 801 kB/s | 107 kB 00:00 2024-09-17T05:25:10.536 INFO:teuthology.orchestra.run.smithi117.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 523 kB/s | 22 kB 00:00 2024-09-17T05:25:10.603 INFO:teuthology.orchestra.run.smithi117.stdout:(51/157): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-17T05:25:10.629 INFO:teuthology.orchestra.run.smithi119.stdout:(72/157): protobuf-3.14.0-13.el9.x86_64.rpm 689 kB/s | 1.0 MB 00:01 2024-09-17T05:25:10.653 INFO:teuthology.orchestra.run.smithi117.stdout:(52/157): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-09-17T05:25:10.679 INFO:teuthology.orchestra.run.smithi117.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-09-17T05:25:10.704 INFO:teuthology.orchestra.run.smithi117.stdout:(54/157): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-09-17T05:25:10.746 INFO:teuthology.orchestra.run.smithi117.stdout:(55/157): libaio-devel-0.3.111-13.el9.x86_64.rp 267 kB/s | 11 kB 00:00 2024-09-17T05:25:10.780 INFO:teuthology.orchestra.run.smithi117.stdout:(56/157): flexiblas-openblas-openmp-3.0.4-8.el9 181 kB/s | 18 kB 00:00 2024-09-17T05:25:10.829 INFO:teuthology.orchestra.run.smithi119.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 437 kB/s | 87 kB 00:00 2024-09-17T05:25:10.830 INFO:teuthology.orchestra.run.smithi117.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 2.0 MB/s | 171 kB 00:00 2024-09-17T05:25:10.872 INFO:teuthology.orchestra.run.smithi117.stdout:(58/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T05:25:10.914 INFO:teuthology.orchestra.run.smithi117.stdout:(59/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-17T05:25:11.056 INFO:teuthology.orchestra.run.smithi117.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-09-17T05:25:11.106 INFO:teuthology.orchestra.run.smithi117.stdout:(61/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 491 kB/s | 24 kB 00:00 2024-09-17T05:25:11.156 INFO:teuthology.orchestra.run.smithi117.stdout:(62/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-09-17T05:25:11.182 INFO:teuthology.orchestra.run.smithi117.stdout:(63/157): libtool-2.4.6-46.el9.x86_64.rpm 2.1 MB/s | 581 kB 00:00 2024-09-17T05:25:11.257 INFO:teuthology.orchestra.run.smithi117.stdout:(64/157): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-09-17T05:25:11.282 INFO:teuthology.orchestra.run.smithi117.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-17T05:25:11.299 INFO:teuthology.orchestra.run.smithi117.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 921 kB/s | 38 kB 00:00 2024-09-17T05:25:11.491 INFO:teuthology.orchestra.run.smithi117.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.6 MB/s | 3.0 MB 00:00 2024-09-17T05:25:11.866 INFO:teuthology.orchestra.run.smithi117.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 4.2 MB/s | 1.6 MB 00:00 2024-09-17T05:25:12.092 INFO:teuthology.orchestra.run.smithi117.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 4.0 MB/s | 910 kB 00:00 2024-09-17T05:25:12.150 INFO:teuthology.orchestra.run.smithi117.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-09-17T05:25:12.279 INFO:teuthology.orchestra.run.smithi119.stdout:(74/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:03 2024-09-17T05:25:12.317 INFO:teuthology.orchestra.run.smithi117.stdout:(71/157): openblas-openmp-0.3.26-2.el9.x86_64.r 4.8 MB/s | 4.9 MB 00:01 2024-09-17T05:25:12.380 INFO:teuthology.orchestra.run.smithi119.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 615 kB/s | 61 kB 00:00 2024-09-17T05:25:12.455 INFO:teuthology.orchestra.run.smithi119.stdout:(76/157): python3-babel-2.9.1-2.el9.noarch.rpm 3.7 MB/s | 6.0 MB 00:01 2024-09-17T05:25:12.509 INFO:teuthology.orchestra.run.smithi117.stdout:(72/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-09-17T05:25:12.567 INFO:teuthology.orchestra.run.smithi117.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-17T05:25:12.580 INFO:teuthology.orchestra.run.smithi119.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-09-17T05:25:12.593 INFO:teuthology.orchestra.run.smithi117.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-09-17T05:25:12.605 INFO:teuthology.orchestra.run.smithi119.stdout:(78/157): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-17T05:25:12.643 INFO:teuthology.orchestra.run.smithi117.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-17T05:25:12.681 INFO:teuthology.orchestra.run.smithi119.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-09-17T05:25:12.714 INFO:teuthology.orchestra.run.smithi119.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-09-17T05:25:12.726 INFO:teuthology.orchestra.run.smithi117.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-17T05:25:12.789 INFO:teuthology.orchestra.run.smithi119.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-17T05:25:12.810 INFO:teuthology.orchestra.run.smithi117.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-17T05:25:12.823 INFO:teuthology.orchestra.run.smithi119.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-09-17T05:25:12.869 INFO:teuthology.orchestra.run.smithi117.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-09-17T05:25:12.936 INFO:teuthology.orchestra.run.smithi117.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 288 kB/s | 19 kB 00:00 2024-09-17T05:25:12.947 INFO:teuthology.orchestra.run.smithi119.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-09-17T05:25:13.041 INFO:teuthology.orchestra.run.smithi117.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-17T05:25:13.048 INFO:teuthology.orchestra.run.smithi119.stdout:(84/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-17T05:25:13.074 INFO:teuthology.orchestra.run.smithi119.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-09-17T05:25:13.094 INFO:teuthology.orchestra.run.smithi117.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 659 kB/s | 35 kB 00:00 2024-09-17T05:25:13.174 INFO:teuthology.orchestra.run.smithi119.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 615 kB/s | 77 kB 00:00 2024-09-17T05:25:13.199 INFO:teuthology.orchestra.run.smithi119.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 337 kB/s | 42 kB 00:00 2024-09-17T05:25:13.291 INFO:teuthology.orchestra.run.smithi119.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-09-17T05:25:13.317 INFO:teuthology.orchestra.run.smithi119.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-17T05:25:13.370 INFO:teuthology.orchestra.run.smithi117.stdout:(82/157): postgresql-server-13.16-1.el9.x86_64. 4.7 MB/s | 5.8 MB 00:01 2024-09-17T05:25:13.409 INFO:teuthology.orchestra.run.smithi119.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-17T05:25:13.434 INFO:teuthology.orchestra.run.smithi119.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-09-17T05:25:13.453 INFO:teuthology.orchestra.run.smithi117.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.0 MB/s | 250 kB 00:00 2024-09-17T05:25:13.512 INFO:teuthology.orchestra.run.smithi117.stdout:(84/157): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-09-17T05:25:13.517 INFO:teuthology.orchestra.run.smithi119.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 495 kB/s | 54 kB 00:00 2024-09-17T05:25:13.621 INFO:teuthology.orchestra.run.smithi117.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 712 kB/s | 77 kB 00:00 2024-09-17T05:25:13.626 INFO:teuthology.orchestra.run.smithi119.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 384 kB/s | 42 kB 00:00 2024-09-17T05:25:13.735 INFO:teuthology.orchestra.run.smithi119.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 408 kB/s | 44 kB 00:00 2024-09-17T05:25:13.762 INFO:teuthology.orchestra.run.smithi117.stdout:(86/157): python3-prettytable-0.7.2-27.el9.noar 298 kB/s | 42 kB 00:00 2024-09-17T05:25:13.838 INFO:teuthology.orchestra.run.smithi117.stdout:(87/157): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 6.0 MB 00:01 2024-09-17T05:25:13.852 INFO:teuthology.orchestra.run.smithi119.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-17T05:25:13.888 INFO:teuthology.orchestra.run.smithi117.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-09-17T05:25:13.913 INFO:teuthology.orchestra.run.smithi117.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-17T05:25:13.969 INFO:teuthology.orchestra.run.smithi119.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 308 kB/s | 36 kB 00:00 2024-09-17T05:25:13.972 INFO:teuthology.orchestra.run.smithi117.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 876 kB/s | 51 kB 00:00 2024-09-17T05:25:13.998 INFO:teuthology.orchestra.run.smithi117.stdout:(91/157): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-09-17T05:25:14.064 INFO:teuthology.orchestra.run.smithi117.stdout:(92/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.3 MB/s | 5.1 MB 00:00 2024-09-17T05:25:14.090 INFO:teuthology.orchestra.run.smithi117.stdout:(93/157): python3-requests-oauthlib-1.3.0-12.el 457 kB/s | 54 kB 00:00 2024-09-17T05:25:14.132 INFO:teuthology.orchestra.run.smithi117.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 623 kB/s | 42 kB 00:00 2024-09-17T05:25:14.136 INFO:teuthology.orchestra.run.smithi119.stdout:(97/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-17T05:25:14.157 INFO:teuthology.orchestra.run.smithi117.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 659 kB/s | 44 kB 00:00 2024-09-17T05:25:14.161 INFO:teuthology.orchestra.run.smithi119.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 247 kB/s | 47 kB 00:00 2024-09-17T05:25:14.190 INFO:teuthology.orchestra.run.smithi117.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-17T05:25:14.216 INFO:teuthology.orchestra.run.smithi117.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-09-17T05:25:14.249 INFO:teuthology.orchestra.run.smithi117.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 812 kB/s | 47 kB 00:00 2024-09-17T05:25:14.253 INFO:teuthology.orchestra.run.smithi119.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-17T05:25:14.278 INFO:teuthology.orchestra.run.smithi119.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 494 kB/s | 58 kB 00:00 2024-09-17T05:25:14.308 INFO:teuthology.orchestra.run.smithi117.stdout:(99/157): socat-1.7.4.1-6.el9.x86_64.rpm 3.2 MB/s | 304 kB 00:00 2024-09-17T05:25:14.333 INFO:teuthology.orchestra.run.smithi117.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 692 kB/s | 58 kB 00:00 2024-09-17T05:25:14.367 INFO:teuthology.orchestra.run.smithi117.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-09-17T05:25:14.370 INFO:teuthology.orchestra.run.smithi119.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 821 kB/s | 96 kB 00:00 2024-09-17T05:25:14.392 INFO:teuthology.orchestra.run.smithi117.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-17T05:25:14.395 INFO:teuthology.orchestra.run.smithi119.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 545 kB/s | 64 kB 00:00 2024-09-17T05:25:14.426 INFO:teuthology.orchestra.run.smithi117.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 763 kB/s | 45 kB 00:00 2024-09-17T05:25:14.479 INFO:teuthology.orchestra.run.smithi119.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 413 kB/s | 45 kB 00:00 2024-09-17T05:25:14.513 INFO:teuthology.orchestra.run.smithi119.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-09-17T05:25:14.604 INFO:teuthology.orchestra.run.smithi119.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 5.8 MB/s | 548 kB 00:00 2024-09-17T05:25:14.663 INFO:teuthology.orchestra.run.smithi119.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 5.2 MB/s | 308 kB 00:00 2024-09-17T05:25:14.676 INFO:teuthology.orchestra.run.smithi117.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-09-17T05:25:14.688 INFO:teuthology.orchestra.run.smithi119.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-17T05:25:14.769 INFO:teuthology.orchestra.run.smithi117.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 5.8 MB/s | 548 kB 00:00 2024-09-17T05:25:14.805 INFO:teuthology.orchestra.run.smithi119.stdout:(108/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-09-17T05:25:14.810 INFO:teuthology.orchestra.run.smithi117.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 7.2 MB/s | 308 kB 00:00 2024-09-17T05:25:14.827 INFO:teuthology.orchestra.run.smithi117.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-17T05:25:14.856 INFO:teuthology.orchestra.run.smithi119.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-09-17T05:25:14.923 INFO:teuthology.orchestra.run.smithi119.stdout:(110/157): postgresql-server-13.16-1.el9.x86_64 979 kB/s | 5.8 MB 00:06 2024-09-17T05:25:14.948 INFO:teuthology.orchestra.run.smithi119.stdout:(111/157): liboath-2.6.7-2.el9.x86_64.rpm 528 kB/s | 49 kB 00:00 2024-09-17T05:25:14.990 INFO:teuthology.orchestra.run.smithi119.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-17T05:25:15.024 INFO:teuthology.orchestra.run.smithi119.stdout:(113/157): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-09-17T05:25:15.082 INFO:teuthology.orchestra.run.smithi119.stdout:(114/157): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-17T05:25:15.085 INFO:teuthology.orchestra.run.smithi117.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-09-17T05:25:15.108 INFO:teuthology.orchestra.run.smithi119.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-17T05:25:15.111 INFO:teuthology.orchestra.run.smithi117.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-17T05:25:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-17T05:25:15.136 INFO:teuthology.orchestra.run.smithi117.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-17T05:25:15.149 INFO:teuthology.orchestra.run.smithi119.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 863 kB/s | 14 kB 00:00 2024-09-17T05:25:15.161 INFO:teuthology.orchestra.run.smithi117.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-17T05:25:15.183 INFO:teuthology.orchestra.run.smithi119.stdout:(118/157): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-09-17T05:25:15.186 INFO:teuthology.orchestra.run.smithi117.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-09-17T05:25:15.228 INFO:teuthology.orchestra.run.smithi117.stdout:(113/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-17T05:25:15.233 INFO:teuthology.orchestra.run.smithi119.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-09-17T05:25:15.262 INFO:teuthology.orchestra.run.smithi117.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-09-17T05:25:15.279 INFO:teuthology.orchestra.run.smithi117.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-09-17T05:25:15.284 INFO:teuthology.orchestra.run.smithi119.stdout:(120/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.8 MB/s | 839 kB 00:00 2024-09-17T05:25:15.313 INFO:teuthology.orchestra.run.smithi117.stdout:(116/157): protobuf-compiler-3.14.0-13.el9.x86_ 973 kB/s | 863 kB 00:00 2024-09-17T05:25:15.317 INFO:teuthology.orchestra.run.smithi119.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-09-17T05:25:15.342 INFO:teuthology.orchestra.run.smithi119.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-17T05:25:15.346 INFO:teuthology.orchestra.run.smithi117.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 479 kB/s | 32 kB 00:00 2024-09-17T05:25:15.372 INFO:teuthology.orchestra.run.smithi117.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 241 kB/s | 14 kB 00:00 2024-09-17T05:25:15.376 INFO:teuthology.orchestra.run.smithi119.stdout:(123/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-09-17T05:25:15.397 INFO:teuthology.orchestra.run.smithi117.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 3.5 MB/s | 172 kB 00:00 2024-09-17T05:25:15.401 INFO:teuthology.orchestra.run.smithi119.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-09-17T05:25:15.427 INFO:teuthology.orchestra.run.smithi119.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-09-17T05:25:15.452 INFO:teuthology.orchestra.run.smithi119.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 711 kB/s | 18 kB 00:00 2024-09-17T05:25:15.477 INFO:teuthology.orchestra.run.smithi119.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-09-17T05:25:15.502 INFO:teuthology.orchestra.run.smithi119.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-09-17T05:25:15.528 INFO:teuthology.orchestra.run.smithi119.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-17T05:25:15.553 INFO:teuthology.orchestra.run.smithi119.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-17T05:25:15.556 INFO:teuthology.orchestra.run.smithi117.stdout:(120/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-17T05:25:15.579 INFO:teuthology.orchestra.run.smithi119.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-17T05:25:15.581 INFO:teuthology.orchestra.run.smithi117.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-09-17T05:25:15.604 INFO:teuthology.orchestra.run.smithi119.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-17T05:25:15.614 INFO:teuthology.orchestra.run.smithi117.stdout:(122/157): python3-cherrypy-18.6.1-2.el9.noarch 1.4 MB/s | 358 kB 00:00 2024-09-17T05:25:15.637 INFO:teuthology.orchestra.run.smithi119.stdout:(133/157): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-17T05:25:15.656 INFO:teuthology.orchestra.run.smithi117.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 3.6 MB/s | 144 kB 00:00 2024-09-17T05:25:15.671 INFO:teuthology.orchestra.run.smithi119.stdout:(134/157): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-09-17T05:25:15.682 INFO:teuthology.orchestra.run.smithi117.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-17T05:25:15.696 INFO:teuthology.orchestra.run.smithi119.stdout:(135/157): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-09-17T05:25:15.707 INFO:teuthology.orchestra.run.smithi117.stdout:(125/157): python3-google-auth-2.34.0-1.el9.noa 1.4 MB/s | 220 kB 00:00 2024-09-17T05:25:15.721 INFO:teuthology.orchestra.run.smithi119.stdout:(136/157): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-17T05:25:15.732 INFO:teuthology.orchestra.run.smithi117.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-09-17T05:25:15.747 INFO:teuthology.orchestra.run.smithi119.stdout:(137/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-09-17T05:25:15.757 INFO:teuthology.orchestra.run.smithi117.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 464 kB/s | 23 kB 00:00 2024-09-17T05:25:15.772 INFO:teuthology.orchestra.run.smithi119.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-17T05:25:15.783 INFO:teuthology.orchestra.run.smithi117.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-17T05:25:15.806 INFO:teuthology.orchestra.run.smithi119.stdout:(139/157): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-09-17T05:25:15.808 INFO:teuthology.orchestra.run.smithi117.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-09-17T05:25:15.831 INFO:teuthology.orchestra.run.smithi119.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-17T05:25:15.850 INFO:teuthology.orchestra.run.smithi117.stdout:(130/157): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-09-17T05:25:15.856 INFO:teuthology.orchestra.run.smithi119.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-09-17T05:25:15.875 INFO:teuthology.orchestra.run.smithi117.stdout:(131/157): python3-logutils-0.3.5-21.el9.noarch 693 kB/s | 46 kB 00:00 2024-09-17T05:25:15.881 INFO:teuthology.orchestra.run.smithi119.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-17T05:25:15.901 INFO:teuthology.orchestra.run.smithi117.stdout:(132/157): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-09-17T05:25:15.923 INFO:teuthology.orchestra.run.smithi119.stdout:(143/157): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-09-17T05:25:15.926 INFO:teuthology.orchestra.run.smithi117.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-17T05:25:15.949 INFO:teuthology.orchestra.run.smithi119.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-09-17T05:25:15.951 INFO:teuthology.orchestra.run.smithi117.stdout:(134/157): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-17T05:25:15.974 INFO:teuthology.orchestra.run.smithi119.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-09-17T05:25:15.976 INFO:teuthology.orchestra.run.smithi117.stdout:(135/157): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-09-17T05:25:15.999 INFO:teuthology.orchestra.run.smithi119.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 400 kB/s | 20 kB 00:00 2024-09-17T05:25:16.001 INFO:teuthology.orchestra.run.smithi117.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-17T05:25:16.025 INFO:teuthology.orchestra.run.smithi119.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-17T05:25:16.027 INFO:teuthology.orchestra.run.smithi117.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-09-17T05:25:16.052 INFO:teuthology.orchestra.run.smithi117.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-17T05:25:16.058 INFO:teuthology.orchestra.run.smithi119.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-09-17T05:25:16.077 INFO:teuthology.orchestra.run.smithi117.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T05:25:16.119 INFO:teuthology.orchestra.run.smithi117.stdout:(140/157): python3-kubernetes-26.1.0-2.el9.noar 3.0 MB/s | 1.0 MB 00:00 2024-09-17T05:25:16.134 INFO:teuthology.orchestra.run.smithi119.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-17T05:25:16.144 INFO:teuthology.orchestra.run.smithi117.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 535 kB/s | 36 kB 00:00 2024-09-17T05:25:16.171 INFO:teuthology.orchestra.run.smithi117.stdout:(142/157): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-17T05:25:16.195 INFO:teuthology.orchestra.run.smithi117.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 986 kB/s | 74 kB 00:00 2024-09-17T05:25:16.220 INFO:teuthology.orchestra.run.smithi117.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-17T05:25:16.246 INFO:teuthology.orchestra.run.smithi117.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T05:25:16.271 INFO:teuthology.orchestra.run.smithi117.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-09-17T05:25:16.296 INFO:teuthology.orchestra.run.smithi117.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-17T05:25:16.322 INFO:teuthology.orchestra.run.smithi117.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-17T05:25:16.372 INFO:teuthology.orchestra.run.smithi117.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-17T05:25:16.384 INFO:teuthology.orchestra.run.smithi119.stdout:(150/157): librados2-19.3.0-4989.gce1b342b.el9. 9.4 MB/s | 3.4 MB 00:00 2024-09-17T05:25:16.443 INFO:teuthology.orchestra.run.smithi119.stdout:(151/157): librbd1-19.3.0-4989.gce1b342b.el9.x8 8.2 MB/s | 3.2 MB 00:00 2024-09-17T05:25:16.539 INFO:teuthology.orchestra.run.smithi117.stdout:(150/157): librados2-19.3.0-4989.gce1b342b.el9. 14 MB/s | 3.4 MB 00:00 2024-09-17T05:25:16.597 INFO:teuthology.orchestra.run.smithi117.stdout:(151/157): NetworkManager-1.51.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-09-17T05:25:16.631 INFO:teuthology.orchestra.run.smithi117.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-09-17T05:25:16.634 INFO:teuthology.orchestra.run.smithi119.stdout:(152/157): NetworkManager-1.51.0-1.el9.x86_64.r 4.6 MB/s | 2.3 MB 00:00 2024-09-17T05:25:16.660 INFO:teuthology.orchestra.run.smithi119.stdout:(153/157): NetworkManager-team-1.51.0-1.el9.x86 188 kB/s | 41 kB 00:00 2024-09-17T05:25:16.681 INFO:teuthology.orchestra.run.smithi117.stdout:(153/157): NetworkManager-tui-1.51.0-1.el9.x86_ 4.9 MB/s | 249 kB 00:00 2024-09-17T05:25:16.751 INFO:teuthology.orchestra.run.smithi117.stdout:(154/157): NetworkManager-libnm-1.51.0-1.el9.x8 8.5 MB/s | 1.8 MB 00:00 2024-09-17T05:25:16.807 INFO:teuthology.orchestra.run.smithi117.stdout:(155/157): librbd1-19.3.0-4989.gce1b342b.el9.x8 6.5 MB/s | 3.2 MB 00:00 2024-09-17T05:25:16.810 INFO:teuthology.orchestra.run.smithi119.stdout:(154/157): NetworkManager-libnm-1.51.0-1.el9.x8 4.2 MB/s | 1.8 MB 00:00 2024-09-17T05:25:16.835 INFO:teuthology.orchestra.run.smithi119.stdout:(155/157): NetworkManager-tui-1.51.0-1.el9.x86_ 1.2 MB/s | 249 kB 00:00 2024-09-17T05:25:16.840 INFO:teuthology.orchestra.run.smithi117.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 8.5 MB/s | 1.4 MB 00:00 2024-09-17T05:25:16.907 INFO:teuthology.orchestra.run.smithi117.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-09-17T05:25:16.913 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-17T05:25:16.913 INFO:teuthology.orchestra.run.smithi117.stdout:Total 20 MB/s | 228 MB 00:11 2024-09-17T05:25:16.969 INFO:teuthology.orchestra.run.smithi119.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 4.4 MB/s | 1.4 MB 00:00 2024-09-17T05:25:17.977 INFO:teuthology.orchestra.run.smithi119.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-09-17T05:25:17.984 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-17T05:25:17.984 INFO:teuthology.orchestra.run.smithi119.stdout:Total 14 MB/s | 228 MB 00:15 2024-09-17T05:25:18.158 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-17T05:25:18.312 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-17T05:25:18.312 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-17T05:25:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-17T05:25:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-17T05:25:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-17T05:25:20.585 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-17T05:25:20.585 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-17T05:25:21.203 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-17T05:25:21.203 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-17T05:25:26.863 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-17T05:25:26.993 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-17T05:25:27.118 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-17T05:25:27.264 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-17T05:25:27.303 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-17T05:25:27.345 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-17T05:25:27.400 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-17T05:25:27.511 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-17T05:25:27.630 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-17T05:25:27.744 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-17T05:25:27.748 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-17T05:25:27.880 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T05:25:28.008 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-17T05:25:28.014 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T05:25:28.048 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T05:25:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-17T05:25:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-17T05:25:28.139 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T05:25:28.169 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-17T05:25:28.169 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T05:25:28.199 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-17T05:25:28.370 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-17T05:25:28.425 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-17T05:25:28.453 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T05:25:28.507 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-17T05:25:28.559 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T05:25:28.595 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T05:25:28.633 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T05:25:28.659 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T05:25:28.688 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T05:25:28.712 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T05:25:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T05:25:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T05:25:28.891 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T05:25:28.907 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T05:25:28.921 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T05:25:28.950 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-17T05:25:29.078 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T05:25:29.107 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-17T05:25:29.169 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-17T05:25:29.197 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T05:25:29.211 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-17T05:25:29.303 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T05:25:29.327 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T05:25:29.332 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T05:25:29.394 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T05:25:29.405 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T05:25:29.416 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T05:25:29.437 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T05:25:29.637 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T05:25:29.642 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T05:25:29.747 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-17T05:25:29.767 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T05:25:29.785 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-17T05:25:29.827 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-17T05:25:29.897 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-17T05:25:29.901 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-17T05:25:29.945 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-17T05:25:29.985 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-17T05:25:30.005 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T05:25:30.029 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-17T05:25:30.061 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-17T05:25:30.074 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T05:25:30.089 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-17T05:25:30.158 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-17T05:25:30.211 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-17T05:25:30.250 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-17T05:25:30.291 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T05:25:30.294 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-17T05:25:30.372 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-17T05:25:30.404 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 34/165 2024-09-17T05:25:30.412 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-17T05:25:30.439 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 35/165 2024-09-17T05:25:30.448 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-17T05:25:30.468 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 36/165 2024-09-17T05:25:30.484 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-17T05:25:30.540 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-17T05:25:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-17T05:25:30.583 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-17T05:25:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-17T05:25:30.650 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-17T05:25:30.657 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-17T05:25:30.702 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-17T05:25:30.734 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-17T05:25:30.770 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-17T05:25:30.790 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-17T05:25:30.840 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-17T05:25:30.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-17T05:25:30.893 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-17T05:25:30.923 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-17T05:25:30.959 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-17T05:25:31.036 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-17T05:25:31.074 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 34/165 2024-09-17T05:25:31.113 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 35/165 2024-09-17T05:25:31.141 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 36/165 2024-09-17T05:25:31.220 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-17T05:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-17T05:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-17T05:25:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-17T05:25:31.439 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-17T05:25:31.468 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-17T05:25:31.482 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-17T05:25:31.512 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-17T05:25:31.533 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-17T05:25:31.557 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 46/165 2024-09-17T05:25:31.600 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-17T05:25:31.753 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-17T05:25:31.827 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-17T05:25:31.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-17T05:25:31.983 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-17T05:25:32.064 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-17T05:25:32.064 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-17T05:25:32.096 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-17T05:25:32.131 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-17T05:25:32.146 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-17T05:25:32.178 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-17T05:25:32.203 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-17T05:25:32.223 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-17T05:25:32.243 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-17T05:25:32.266 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-17T05:25:32.281 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 46/165 2024-09-17T05:25:32.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-17T05:25:32.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-17T05:25:32.444 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-17T05:25:32.485 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-17T05:25:32.489 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-17T05:25:32.526 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-17T05:25:32.553 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-17T05:25:32.645 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-17T05:25:32.709 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-17T05:25:32.782 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-17T05:25:32.797 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-17T05:25:32.821 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-17T05:25:32.836 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-17T05:25:32.866 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-17T05:25:32.877 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-17T05:25:32.901 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-17T05:25:32.912 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-17T05:25:32.925 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-17T05:25:32.948 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-17T05:25:32.991 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-17T05:25:33.016 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-17T05:25:33.033 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-17T05:25:33.143 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-17T05:25:33.170 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-17T05:25:33.174 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-17T05:25:33.204 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-17T05:25:33.237 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-17T05:25:33.275 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-17T05:25:33.307 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-17T05:25:33.319 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-17T05:25:33.340 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-17T05:25:33.374 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-17T05:25:33.377 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-17T05:25:33.408 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-17T05:25:33.441 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-17T05:25:33.481 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-17T05:25:33.484 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-17T05:25:33.519 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-17T05:25:33.558 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-17T05:25:33.570 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-17T05:25:33.584 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-17T05:25:33.609 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-17T05:25:33.614 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-17T05:25:33.646 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-17T05:25:33.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-17T05:25:33.750 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-17T05:25:33.805 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-17T05:25:33.810 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-17T05:25:33.849 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-17T05:25:33.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-17T05:25:33.912 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-17T05:25:33.959 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-17T05:25:33.999 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-17T05:25:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-17T05:25:34.064 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-17T05:25:34.096 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-17T05:25:34.138 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-17T05:25:34.173 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-17T05:25:34.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-17T05:25:34.303 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-17T05:25:34.343 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-17T05:25:34.464 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-17T05:25:34.519 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-17T05:25:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-17T05:25:34.568 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-17T05:25:34.611 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-17T05:25:34.718 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-17T05:25:34.774 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-17T05:25:34.819 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-17T05:25:34.907 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-17T05:25:34.946 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-17T05:25:35.130 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-17T05:25:35.177 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-17T05:25:35.254 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-17T05:25:35.255 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-17T05:25:35.307 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-17T05:25:35.349 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-17T05:25:35.459 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-17T05:25:35.513 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-17T05:25:35.558 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-17T05:25:35.662 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-17T05:25:35.693 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-17T05:25:35.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-17T05:25:35.899 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-17T05:25:35.910 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-17T05:25:35.977 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-17T05:25:36.036 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-17T05:25:36.630 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-17T05:25:36.756 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-17T05:25:37.289 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-17T05:25:37.366 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-17T05:25:37.393 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-17T05:25:37.426 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-17T05:25:37.752 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-17T05:25:37.808 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T05:25:37.900 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T05:25:37.930 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 103/165 2024-09-17T05:25:37.969 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-17T05:25:37.992 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-17T05:25:38.070 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-17T05:25:38.098 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-17T05:25:38.130 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-17T05:25:38.456 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-17T05:25:38.464 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-17T05:25:38.494 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T05:25:38.504 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T05:25:38.565 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T05:25:38.593 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T05:25:38.596 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 107/165 2024-09-17T05:25:38.623 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 103/165 2024-09-17T05:25:38.667 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-17T05:25:39.175 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-17T05:25:39.205 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T05:25:39.259 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T05:25:39.281 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 107/165 2024-09-17T05:25:40.914 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T05:25:40.951 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T05:25:41.508 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T05:25:41.521 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T05:25:41.544 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T05:25:41.571 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T05:25:41.674 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T05:25:41.674 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T05:25:41.674 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T05:25:41.674 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:41.714 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T05:25:42.114 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T05:25:42.156 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T05:25:42.256 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T05:25:42.256 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T05:25:42.256 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T05:25:42.256 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:42.290 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-09-17T05:25:53.312 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:53.491 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T05:25:53.513 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T05:25:53.513 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:53.513 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T05:25:53.513 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:53.920 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T05:25:53.920 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-09-17T05:25:53.920 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-09-17T05:25:53.920 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-09-17T05:25:53.921 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-09-17T05:25:53.921 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-09-17T05:25:53.921 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-09-17T05:25:53.921 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-09-17T05:25:53.921 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:54.093 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T05:25:54.111 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T05:25:54.111 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:54.111 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T05:25:54.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:55.102 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T05:25:55.130 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T05:25:55.130 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:55.130 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T05:25:55.131 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T05:25:55.131 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T05:25:55.131 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:55.361 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T05:25:55.385 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T05:25:55.385 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:55.385 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T05:25:55.385 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T05:25:55.386 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T05:25:55.386 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:55.688 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T05:25:55.715 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T05:25:55.715 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:55.715 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T05:25:55.715 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T05:25:55.715 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T05:25:55.715 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:55.853 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T05:25:55.877 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T05:25:55.877 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:55.878 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T05:25:55.878 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T05:25:55.878 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T05:25:55.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:25:55.946 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T05:25:55.970 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T05:25:55.970 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:55.970 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T05:25:55.970 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T05:25:55.971 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T05:25:55.971 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:55.983 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-17T05:25:56.041 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-17T05:25:56.104 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-17T05:25:56.294 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-17T05:25:56.329 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-17T05:25:56.407 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-17T05:25:56.439 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T05:25:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T05:25:56.463 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:25:56.463 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T05:25:56.463 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T05:25:56.463 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T05:25:56.463 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:25:56.497 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-17T05:25:56.569 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-17T05:25:56.626 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-17T05:25:56.633 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-17T05:25:56.671 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-17T05:25:56.690 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-17T05:25:56.703 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-17T05:25:56.731 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-17T05:25:56.888 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-17T05:25:56.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-17T05:25:57.010 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-17T05:25:57.100 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-17T05:25:57.187 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T05:25:57.213 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T05:25:57.241 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-17T05:25:57.245 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 127/165 2024-09-17T05:25:57.290 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-17T05:25:57.311 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 128/165 2024-09-17T05:25:57.330 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-17T05:25:57.344 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T05:25:57.366 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-17T05:25:57.729 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T05:25:57.756 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T05:25:57.789 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 127/165 2024-09-17T05:25:57.856 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 128/165 2024-09-17T05:25:57.888 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T05:26:00.515 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T05:26:00.567 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T05:26:01.086 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T05:26:01.144 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T05:26:01.527 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T05:26:01.558 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T05:26:01.674 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T05:26:01.833 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 132/165 2024-09-17T05:26:01.860 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T05:26:01.884 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T05:26:01.885 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:01.885 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T05:26:01.885 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T05:26:01.885 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T05:26:01.885 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:01.936 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T05:26:01.948 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T05:26:02.046 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 135/165 2024-09-17T05:26:02.094 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 136/165 2024-09-17T05:26:02.103 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T05:26:02.135 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T05:26:02.254 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T05:26:02.421 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 132/165 2024-09-17T05:26:02.454 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T05:26:02.478 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T05:26:02.479 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:02.479 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T05:26:02.479 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T05:26:02.479 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T05:26:02.479 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:02.530 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T05:26:02.541 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T05:26:02.640 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 135/165 2024-09-17T05:26:02.687 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 136/165 2024-09-17T05:26:03.063 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-17T05:26:03.097 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T05:26:03.121 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T05:26:03.121 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.121 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T05:26:03.121 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T05:26:03.121 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T05:26:03.121 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:03.164 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T05:26:03.188 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T05:26:03.188 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.188 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T05:26:03.188 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:03.240 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T05:26:03.263 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T05:26:03.263 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.263 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-17T05:26:03.263 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T05:26:03.264 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T05:26:03.264 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:03.582 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T05:26:03.606 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T05:26:03.606 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.606 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T05:26:03.606 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T05:26:03.606 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T05:26:03.606 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:03.665 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-17T05:26:03.699 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T05:26:03.722 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T05:26:03.722 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.722 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T05:26:03.723 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T05:26:03.723 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T05:26:03.723 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:03.766 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T05:26:03.789 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T05:26:03.790 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.790 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T05:26:03.790 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:03.842 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T05:26:03.866 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T05:26:03.866 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:03.866 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-17T05:26:03.866 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T05:26:03.866 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T05:26:03.866 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:04.185 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T05:26:04.209 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T05:26:04.209 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T05:26:04.209 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T05:26:04.209 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T05:26:04.210 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T05:26:04.210 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:07.628 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 142/165 2024-09-17T05:26:07.669 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 143/165 2024-09-17T05:26:07.710 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 144/165 2024-09-17T05:26:07.742 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-17T05:26:07.760 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-17T05:26:07.951 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 147/165 2024-09-17T05:26:07.999 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-17T05:26:08.029 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 149/165 2024-09-17T05:26:08.059 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-17T05:26:08.113 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-17T05:26:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 142/165 2024-09-17T05:26:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 143/165 2024-09-17T05:26:08.270 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 144/165 2024-09-17T05:26:08.302 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-17T05:26:08.329 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-17T05:26:08.516 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 147/165 2024-09-17T05:26:08.560 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-17T05:26:08.590 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 149/165 2024-09-17T05:26:08.620 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-17T05:26:08.674 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-17T05:26:08.754 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T05:26:08.794 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T05:26:08.816 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T05:26:09.308 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T05:26:09.347 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T05:26:09.370 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T05:26:09.630 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 153/165 2024-09-17T05:26:09.724 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-17T05:26:09.772 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-17T05:26:09.862 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-17T05:26:09.894 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-17T05:26:09.920 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-17T05:26:09.953 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-17T05:26:09.953 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T05:26:10.037 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T05:26:10.037 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T05:26:10.121 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T05:26:10.146 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-17T05:26:10.152 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T05:26:10.152 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T05:26:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 153/165 2024-09-17T05:26:10.309 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-17T05:26:10.348 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T05:26:10.349 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T05:26:10.350 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-17T05:26:10.433 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T05:26:10.440 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-17T05:26:10.479 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-17T05:26:10.498 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T05:26:10.504 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-17T05:26:10.539 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-17T05:26:10.539 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T05:26:10.631 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T05:26:10.631 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T05:26:10.707 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T05:26:10.732 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-17T05:26:10.738 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T05:26:10.738 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T05:26:10.908 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T05:26:10.909 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T05:26:11.001 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T05:26:11.065 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 1/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 3/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 4/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 5/165 2024-09-17T05:26:16.865 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 6/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 7/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 11/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 12/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 13/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 15/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 16/165 2024-09-17T05:26:16.866 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 17/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 18/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 19/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 20/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 21/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 22/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 23/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 24/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 25/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 26/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 27/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 28/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 29/165 2024-09-17T05:26:16.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 30/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 31/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 32/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 33/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 34/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 35/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 36/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 37/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 38/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 39/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-17T05:26:16.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-17T05:26:16.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-17T05:26:16.870 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-17T05:26:16.871 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-17T05:26:16.872 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-17T05:26:16.873 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-17T05:26:16.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-17T05:26:16.875 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-17T05:26:16.876 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 150/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 152/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-17T05:26:16.877 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-17T05:26:16.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 1/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 3/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 4/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 5/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 6/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 7/165 2024-09-17T05:26:17.375 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 11/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 12/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 13/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 15/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 16/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 17/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 18/165 2024-09-17T05:26:17.376 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 19/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 20/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 21/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 22/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 23/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 24/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 25/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 26/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 27/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 28/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 29/165 2024-09-17T05:26:17.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 30/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 31/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 32/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 33/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 34/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 35/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 36/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 37/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 38/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 39/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-17T05:26:17.378 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-17T05:26:17.379 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-17T05:26:17.380 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-17T05:26:17.381 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-17T05:26:17.382 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-17T05:26:17.383 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-17T05:26:17.384 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-17T05:26:17.385 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-17T05:26:17.386 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 150/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-17T05:26:17.387 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 152/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-17T05:26:17.388 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.390 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: automake-1.16.2-8.el9.noarch 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: bison-3.7.4-5.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.391 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.392 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: flex-2.6.4-9.el9.x86_64 2024-09-17T05:26:19.393 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.394 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T05:26:19.395 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-17T05:26:19.396 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-13.16-1.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T05:26:19.398 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T05:26:19.399 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T05:26:19.400 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.401 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T05:26:19.402 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-17T05:26:19.403 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-17T05:26:19.404 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T05:26:19.404 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-17T05:26:19.404 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:19.404 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-17T05:26:19.847 DEBUG:teuthology.parallel:result is None 2024-09-17T05:26:19.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T05:26:19.963 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T05:26:19.964 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.2-8.el9.noarch 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.7.4-5.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.966 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.4-9.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T05:26:19.967 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.968 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-17T05:26:19.969 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-13.16-1.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T05:26:19.970 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T05:26:19.971 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T05:26:19.972 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T05:26:19.973 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T05:26:19.974 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T05:26:19.975 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T05:26:19.976 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-17T05:26:19.977 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:19.977 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-17T05:26:20.399 DEBUG:teuthology.parallel:result is None 2024-09-17T05:26:20.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:26:20.549 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T05:26:20.572 INFO:teuthology.orchestra.run.smithi117.stdout:19.3.0-4989.gce1b342b.el9 2024-09-17T05:26:20.573 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4989.gce1b342b.el9 2024-09-17T05:26:20.573 INFO:teuthology.task.install:The correct ceph version 19.3.0-4989.gce1b342b is installed. 2024-09-17T05:26:20.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:26:20.708 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T05:26:20.730 INFO:teuthology.orchestra.run.smithi119.stdout:19.3.0-4989.gce1b342b.el9 2024-09-17T05:26:20.730 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4989.gce1b342b.el9 2024-09-17T05:26:20.730 INFO:teuthology.task.install:The correct ceph version 19.3.0-4989.gce1b342b is installed. 2024-09-17T05:26:20.732 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T05:26:20.732 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:20.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T05:26:20.761 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:20.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T05:26:20.797 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T05:26:20.797 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:20.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T05:26:20.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T05:26:20.892 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:20.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T05:26:20.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T05:26:20.982 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T05:26:20.982 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:20.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T05:26:21.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T05:26:21.072 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:21.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T05:26:21.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T05:26:21.161 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T05:26:21.161 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:21.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T05:26:21.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T05:26:21.250 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:21.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T05:26:21.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T05:26:21.340 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T05:26:21.424 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T05:26:21.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-09-17T05:26:21.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2024-09-17T05:26:21.451 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T05:26:21.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T05:26:21.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T05:26:21.516 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T05:26:21.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T05:26:21.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T05:26:21.587 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-17T05:26:21.587 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-17T05:26:21.587 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-17T05:26:21.587 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:21.587 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-17T05:26:21.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-17T05:26:21.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-17T05:26:21.767 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:21.767 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-17T05:26:21.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-17T05:26:21.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-17T05:26:21.922 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T05:26:21.922 INFO:tasks.ceph:config {'conf': {'client': {'client_shutdown_timeout': 120}, 'global': {'ms die on skipped message': False}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T05:26:21.922 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618', 'branch': 'wip-pdonnell-testing-20240916.200549-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7908618', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240916.200549-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'global': {'ms die on skipped message': False}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240916.200549-debug', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 476, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'sleep_before_teardown': 0, 'subset': '320/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240916.200549-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOtGJUVP3tsnt+XrOJimpa8Dq+T2MMavQtRhjpphZ5mfNej7vtLfRjhCIT+/trd4eM1sedNe258MdPbDAkAQMY=', 'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpQUKABIPRs1JUblvm15W4ZuknnO0I6pA32xlSAfhqx6vTWx1NN9Lq6vx1OOa5ktYYLpx3w4HDr3XbSGsgw2fg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'global': {'ms die on skipped message': False}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_02:09:51', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T05:26:21.923 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T05:26:21.925 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T05:26:21.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T05:26:21.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T05:26:22.036 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:22.036 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T05:26:22.058 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T05:26:22.059 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-09-17T05:26:22.114 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T05:26:22.114 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:22.114 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T05:26:22.114 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:22.114 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:22.115 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-17 05:26:10.845010343 +0000 2024-09-17T05:26:22.115 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-17 05:24:11.671144277 +0000 2024-09-17T05:26:22.115 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-17 05:24:11.671144277 +0000 2024-09-17T05:26:22.115 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-17 05:24:11.671144277 +0000 2024-09-17T05:26:22.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T05:26:22.178 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-17T05:26:22.178 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-17T05:26:22.178 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000191278 s, 2.7 MB/s 2024-09-17T05:26:22.179 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T05:26:22.236 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-09-17T05:26:22.292 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T05:26:22.292 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:22.292 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T05:26:22.292 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:22.292 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:22.293 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-17 05:26:10.845010343 +0000 2024-09-17T05:26:22.293 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-17 05:24:12.006137386 +0000 2024-09-17T05:26:22.293 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-17 05:24:12.006137386 +0000 2024-09-17T05:26:22.293 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-17 05:24:12.006137386 +0000 2024-09-17T05:26:22.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T05:26:22.355 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-17T05:26:22.355 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-17T05:26:22.355 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000169008 s, 3.0 MB/s 2024-09-17T05:26:22.356 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T05:26:22.412 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-09-17T05:26:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T05:26:22.467 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:22.467 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-17T05:26:22.467 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:22.467 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:22.468 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-17 05:26:10.846010323 +0000 2024-09-17T05:26:22.468 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-17 05:24:12.331130699 +0000 2024-09-17T05:26:22.468 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-17 05:24:12.331130699 +0000 2024-09-17T05:26:22.468 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-17 05:24:12.331130699 +0000 2024-09-17T05:26:22.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T05:26:22.530 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-17T05:26:22.530 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-17T05:26:22.531 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000271977 s, 1.9 MB/s 2024-09-17T05:26:22.532 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T05:26:22.588 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-17 05:26:10.846010323 +0000 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-17 05:24:12.656124013 +0000 2024-09-17T05:26:22.643 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-17 05:24:12.656124013 +0000 2024-09-17T05:26:22.644 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-17 05:24:12.656124013 +0000 2024-09-17T05:26:22.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T05:26:22.706 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-17T05:26:22.706 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-17T05:26:22.706 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000167969 s, 3.0 MB/s 2024-09-17T05:26:22.707 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T05:26:22.763 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:22.763 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:22.763 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T05:26:22.781 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T05:26:22.781 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-09-17T05:26:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T05:26:22.840 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-17 05:26:11.437870888 +0000 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-17 05:24:23.767188826 +0000 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-17 05:24:23.767188826 +0000 2024-09-17T05:26:22.841 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-17 05:24:23.767188826 +0000 2024-09-17T05:26:22.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T05:26:22.906 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-17T05:26:22.906 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-17T05:26:22.906 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000272931 s, 1.9 MB/s 2024-09-17T05:26:22.907 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T05:26:22.965 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-09-17T05:26:23.022 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T05:26:23.022 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:23.022 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T05:26:23.022 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:23.022 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:23.022 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-17 05:26:11.437870888 +0000 2024-09-17T05:26:23.023 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-17 05:24:24.119179908 +0000 2024-09-17T05:26:23.023 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-17 05:24:24.119179908 +0000 2024-09-17T05:26:23.023 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-17 05:24:24.119179908 +0000 2024-09-17T05:26:23.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T05:26:23.088 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-17T05:26:23.088 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-17T05:26:23.088 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000223141 s, 2.3 MB/s 2024-09-17T05:26:23.089 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T05:26:23.146 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-09-17T05:26:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-17 05:26:11.437870888 +0000 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-17 05:24:24.494170408 +0000 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-17 05:24:24.494170408 +0000 2024-09-17T05:26:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-17 05:24:24.494170408 +0000 2024-09-17T05:26:23.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T05:26:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-17T05:26:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-17T05:26:23.267 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000158108 s, 3.2 MB/s 2024-09-17T05:26:23.268 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T05:26:23.324 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-09-17T05:26:23.379 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-17 05:26:11.438870863 +0000 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-17 05:24:24.826161996 +0000 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-17 05:24:24.826161996 +0000 2024-09-17T05:26:23.380 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-17 05:24:24.826161996 +0000 2024-09-17T05:26:23.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T05:26:23.443 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-17T05:26:23.443 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-17T05:26:23.443 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000294029 s, 1.7 MB/s 2024-09-17T05:26:23.444 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T05:26:23.500 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:23.501 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-17T05:26:23.501 INFO:tasks.ceph:Generating config... 2024-09-17T05:26:23.502 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-09-17T05:26:23.502 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-17T05:26:23.502 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-09-17T05:26:23.502 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-09-17T05:26:23.502 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T05:26:23.502 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T05:26:23.503 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T05:26:23.503 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T05:26:23.503 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T05:26:23.503 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-09-17T05:26:23.503 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T05:26:23.504 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T05:26:23.504 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T05:26:23.504 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-17T05:26:23.504 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T05:26:23.504 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-17T05:26:23.504 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T05:26:23.504 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T05:26:23.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T05:26:23.566 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T05:26:23.569 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T05:26:23.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T05:26:23.649 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.117'), ('mon.b', '172.21.15.119'), ('mon.c', '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]')] 2024-09-17T05:26:23.649 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.117,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client_shutdown_timeout': 120}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'mds oft prefetch dirfrags': True, 'mds_sleep_rank_change': 5000000.0}} 2024-09-17T05:26:23.650 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:23.650 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T05:26:23.704 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.117 --add b 172.21.15.119 --addv c '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: generated fsid 8527add3-c46f-4f7f-86e8-5cd706a0043e 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = quincy 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 0 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:fsid 8527add3-c46f-4f7f-86e8-5cd706a0043e 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:last_changed 2024-09-17T05:26:23.802112+0000 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-09-17T05:26:23.802112+0000 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:min_mon_release 17 (quincy) 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:election_strategy: 1 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-09-17T05:26:23.802 INFO:teuthology.orchestra.run.smithi117.stdout:1: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-09-17T05:26:23.803 INFO:teuthology.orchestra.run.smithi117.stdout:2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-17T05:26:23.803 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T05:26:23.804 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T05:26:23.819 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8527add3-c46f-4f7f-86e8-5cd706a0043e... 2024-09-17T05:26:23.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T05:26:23.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T05:26:23.901 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-09-17T05:26:23.901 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-09-17T05:26:23.901 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.902 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-09-17T05:26:23.903 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-09-17T05:26:23.904 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.117,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]" 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on skipped message = False 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 8527add3-c46f-4f7f-86e8-5cd706a0043e 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.905 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-09-17T05:26:23.906 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.907 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore compression mode = aggressive 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 20 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 20 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-17T05:26:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: filestore flush min = 0 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: osd blocked scrub grace period = 3600 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: osd max markdown count = 1000 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-17T05:26:23.909 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-09-17T05:26:23.910 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T05:26:23.911 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: mon down mkfs grace = 300 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T05:26:23.912 INFO:teuthology.orchestra.run.smithi117.stdout: client_shutdown_timeout = 120 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.c] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.e] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.d] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.f] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout:[mds] 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout: mds oft prefetch dirfrags = True 2024-09-17T05:26:23.913 INFO:teuthology.orchestra.run.smithi117.stdout: mds_sleep_rank_change = 5000000.0 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.914 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T05:26:23.915 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T05:26:23.916 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-17T05:26:23.917 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.117,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]" 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on skipped message = False 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 8527add3-c46f-4f7f-86e8-5cd706a0043e 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-17T05:26:23.918 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug trim objects = true 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.919 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore compression mode = aggressive 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-17T05:26:23.920 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: filestore flush min = 0 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: osd blocked scrub grace period = 3600 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: osd max markdown count = 1000 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-17T05:26:23.921 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-17T05:26:23.922 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.923 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: mon down mkfs grace = 300 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T05:26:23.924 INFO:teuthology.orchestra.run.smithi119.stdout: client_shutdown_timeout = 120 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.e] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.f] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout: mds oft prefetch dirfrags = True 2024-09-17T05:26:23.925 INFO:teuthology.orchestra.run.smithi119.stdout: mds_sleep_rank_change = 5000000.0 2024-09-17T05:26:23.927 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T05:26:23.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T05:26:24.030 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T05:26:24.030 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:24.030 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T05:26:24.045 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:24.046 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T05:26:24.101 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:26:24.101 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:24.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T05:26:24.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T05:26:24.174 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:24.174 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:24.230 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:26:24.231 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:24.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T05:26:24.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T05:26:24.266 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:24.267 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:24.322 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T05:26:24.323 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T05:26:24.323 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T05:26:24.384 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T05:26:24.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T05:26:24.454 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T05:26:24.457 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T05:26:24.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-17T05:26:24.518 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-17T05:26:24.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-17T05:26:24.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-17T05:26:24.645 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-17T05:26:24.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-17T05:26:24.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-09-17T05:26:24.773 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-09-17T05:26:24.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-09-17T05:26:24.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-17T05:26:24.863 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-17T05:26:25.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-17T05:26:25.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-17T05:26:25.220 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-17T05:26:25.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-17T05:26:25.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-09-17T05:26:25.348 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-09-17T05:26:25.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-09-17T05:26:25.376 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T05:26:25.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T05:26:25.428 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T05:26:25.441 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T05:26:25.441 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-17T05:26:25.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T05:26:25.465 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:25.465 INFO:tasks.ceph:role: osd.0 2024-09-17T05:26:25.465 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:26:25.466 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T05:26:25.537 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:25.537 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:25.538 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:26.021 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-17T05:26:26.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-17T05:26:26.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T05:26:26.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T05:26:26.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T05:26:26.234 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:26.234 INFO:tasks.ceph:role: osd.1 2024-09-17T05:26:26.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:26:26.234 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T05:26:26.306 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:26.760 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-17T05:26:26.762 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-17T05:26:26.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T05:26:26.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T05:26:26.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T05:26:26.970 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:26.971 INFO:tasks.ceph:role: osd.2 2024-09-17T05:26:26.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:26:26.971 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:27.043 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:27.539 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-17T05:26:27.541 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-17T05:26:27.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T05:26:27.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T05:26:27.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T05:26:27.733 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:27.733 INFO:tasks.ceph:role: osd.3 2024-09-17T05:26:27.733 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:26:27.734 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:27.806 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:28.326 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-17T05:26:28.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-17T05:26:28.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-17T05:26:28.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T05:26:28.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:28.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:28.606+0000 7fc55e0b6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T05:26:28.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:28.611+0000 7fc55e0b6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T05:26:28.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:28.611+0000 7fc55e0b6640 -1 bdev(0x5639fb6e3900 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:28.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:28.611+0000 7fc55e0b6640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T05:26:32.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T05:26:32.046 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:32.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:32.147+0000 7f0039c0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T05:26:32.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:32.148+0000 7f0039c0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T05:26:32.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:32.148+0000 7f0039c0c640 -1 bdev(0x56182bc01900 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:32.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:32.148+0000 7f0039c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T05:26:35.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T05:26:35.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:35.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:35.437+0000 7f4ea53b9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T05:26:35.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:35.438+0000 7f4ea53b9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T05:26:35.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:35.438+0000 7f4ea53b9640 -1 bdev(0x55b9c4b11900 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:35.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:35.438+0000 7f4ea53b9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T05:26:38.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T05:26:38.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:38.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:38.486+0000 7f01f81b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T05:26:38.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:38.486+0000 7f01f81b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T05:26:38.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:38.486+0000 7f01f81b2640 -1 bdev(0x5613e8bfb900 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:38.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:26:38.486+0000 7f01f81b2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T05:26:41.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T05:26:41.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T05:26:41.462 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:41.463 INFO:tasks.ceph:role: osd.4 2024-09-17T05:26:41.463 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:26:41.463 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T05:26:41.539 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:41.539 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:41.539 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:41.539 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:41.539 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:41.539 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:41.540 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:41.540 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:41.540 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:41.540 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:42.024 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-17T05:26:42.026 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-17T05:26:42.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-17T05:26:42.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T05:26:42.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T05:26:42.239 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:42.240 INFO:tasks.ceph:role: osd.5 2024-09-17T05:26:42.240 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:26:42.240 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:42.314 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:42.769 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-17T05:26:42.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-17T05:26:42.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-17T05:26:42.844 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T05:26:42.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-17T05:26:42.982 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:42.982 INFO:tasks.ceph:role: osd.6 2024-09-17T05:26:42.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:26:42.982 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T05:26:43.054 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:43.054 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:43.055 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:43.543 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-17T05:26:43.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-17T05:26:43.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-17T05:26:43.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-17T05:26:43.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-17T05:26:43.729 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T05:26:43.729 INFO:tasks.ceph:role: osd.7 2024-09-17T05:26:43.729 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:26:43.729 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T05:26:43.802 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T05:26:44.310 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-09-17T05:26:44.312 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-17T05:26:44.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-17T05:26:44.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-17T05:26:44.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:44.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:44.588+0000 7fa79e89c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T05:26:44.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:44.590+0000 7fa79e89c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T05:26:44.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:44.590+0000 7fa79e89c640 -1 bdev(0x55fdc302f900 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:44.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:44.590+0000 7fa79e89c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T05:26:48.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T05:26:48.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:48.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:48.138+0000 7f0638d02640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T05:26:48.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:48.138+0000 7f0638d02640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T05:26:48.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:48.139+0000 7f0638d02640 -1 bdev(0x55efb1d77900 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:48.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:48.139+0000 7f0638d02640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T05:26:51.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T05:26:51.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:51.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:51.181+0000 7fe35d534640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-17T05:26:51.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:51.182+0000 7fe35d534640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-17T05:26:51.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:51.182+0000 7fe35d534640 -1 bdev(0x5609d4373900 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:51.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:51.182+0000 7fe35d534640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-17T05:26:51.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:26:51.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:26:54.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T05:26:54.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:26:54.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:54.227+0000 7fcd85945640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-17T05:26:54.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:54.228+0000 7fcd85945640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-17T05:26:54.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:54.228+0000 7fcd85945640 -1 bdev(0x5601614c5900 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-17T05:26:54.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:26:54.228+0000 7fcd85945640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-17T05:26:57.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T05:26:57.907 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T05:26:57.908 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:57.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-17T05:26:57.938 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:57.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-17T05:26:58.007 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-17T05:26:58.075 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-09-17T05:26:58.143 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T05:26:58.209 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T05:26:58.276 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T05:26:58.343 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T05:26:58.409 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T05:26:58.439 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-17T05:26:58.508 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-17T05:26:58.576 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-09-17T05:26:58.643 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T05:26:58.709 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T05:26:58.775 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-17T05:26:58.841 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:26:58.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-17T05:26:58.906 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:26:58.906 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T05:26:58.924 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T05:26:58.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T05:26:58.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T05:26:58.990 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2024-09-17T05:26:58.990 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAAE+lmIAHpFhAA3c/wVnI5j7BfLpAE51lgLQ== 2024-09-17T05:26:58.990 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a] 2024-09-17T05:26:58.990 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAAE+lm8iTpHhAA73OEu8+8UwyqynEo2/I1Ug== 2024-09-17T05:26:58.990 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.c] 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAAE+lmRwB7JhAAQBgubJbUxE0VLHJ9dLVdcg== 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.e] 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAAE+lmvvgcLhAA1Vo58WArxI6hUIzX2VlrOw== 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAEE+lmteI2JBAA5thvAMdPZvrxrWHY80f/yw== 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAIE+lmL1LnCBAA8XmIwS9O5+jzGK0iHAL+Kw== 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQALE+lmX+wmGhAA3j8xdeALkrGPAmNsg7h7BA== 2024-09-17T05:26:58.991 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAOE+lmwrYFHRAAh4Ykou0JEj2UT34NgCrWRw== 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAAE+lm4dUcGxAAMjFHqVfjUHgfq9r0LrM4pg== 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b] 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAAE+lm0Yl6MxAAigm2wFjyB9mfFxtcnwdc4A== 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.d] 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABE+lm79wmDRAAIyM6ICfNCXUSifx9zOerFg== 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.f] 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABE+lmvbHRFBAA6Qr16kDF/7/dUgBNog78jg== 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-09-17T05:26:58.992 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAUE+lmRNofIxAARjTQxIbQ41wjjXE3mNZ/UA== 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAYE+lmi6JfCBAAq+tZyle3C4E4dPxFCgmfRA== 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAbE+lmuJrxChAAcEbNOR39EN2/chwInRjPjw== 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAeE+lmI6aiDRAA+omix5kgZPCmzQGyM2qtXA== 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABE+lmzOiLGRAA9RHVVqdWZzw3h2Ej6W4YYw== 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2024-09-17T05:26:58.993 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAAE+lmIAHpFhAA3c/wVnI5j7BfLpAE51lgLQ== 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAAE+lm8iTpHhAA73OEu8+8UwyqynEo2/I1Ug== 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAAE+lmRwB7JhAAQBgubJbUxE0VLHJ9dLVdcg== 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.e] 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAAE+lmvvgcLhAA1Vo58WArxI6hUIzX2VlrOw== 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAEE+lmteI2JBAA5thvAMdPZvrxrWHY80f/yw== 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAIE+lmL1LnCBAA8XmIwS9O5+jzGK0iHAL+Kw== 2024-09-17T05:26:58.994 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQALE+lmX+wmGhAA3j8xdeALkrGPAmNsg7h7BA== 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAOE+lmwrYFHRAAh4Ykou0JEj2UT34NgCrWRw== 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAAE+lm4dUcGxAAMjFHqVfjUHgfq9r0LrM4pg== 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAAE+lm0Yl6MxAAigm2wFjyB9mfFxtcnwdc4A== 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQABE+lm79wmDRAAIyM6ICfNCXUSifx9zOerFg== 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.f] 2024-09-17T05:26:58.995 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQABE+lmvbHRFBAA6Qr16kDF/7/dUgBNog78jg== 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAUE+lmRNofIxAARjTQxIbQ41wjjXE3mNZ/UA== 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAYE+lmi6JfCBAAq+tZyle3C4E4dPxFCgmfRA== 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAbE+lmuJrxChAAcEbNOR39EN2/chwInRjPjw== 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAeE+lmI6aiDRAA+omix5kgZPCmzQGyM2qtXA== 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-09-17T05:26:58.996 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQABE+lmzOiLGRAA9RHVVqdWZzw3h2Ej6W4YYw== 2024-09-17T05:26:58.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T05:26:59.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T05:26:59.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.306 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.559 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T05:26:59.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T05:26:59.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T05:26:59.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:26:59.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:27:00.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:27:00.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T05:27:00.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T05:27:00.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T05:27:00.141 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T05:27:00.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T05:27:00.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T05:27:00.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T05:27:00.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T05:27:00.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T05:27:00.426 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T05:27:00.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T05:27:00.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T05:27:00.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T05:27:00.645 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:27:00.647 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T05:27:00.700 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T05:27:00.701 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T05:27:00.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T05:27:00.704 INFO:tasks.ceph.mon.a:Started 2024-09-17T05:27:00.705 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T05:27:00.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T05:27:00.743 INFO:tasks.ceph.mon.b:Started 2024-09-17T05:27:00.743 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T05:27:00.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T05:27:00.745 INFO:tasks.ceph.mon.c:Started 2024-09-17T05:27:00.745 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T05:27:00.745 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-17T05:27:00.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-17T05:27:00.748 INFO:tasks.ceph.mgr.y:Started 2024-09-17T05:27:00.748 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T05:27:00.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T05:27:00.752 INFO:tasks.ceph.mgr.x:Started 2024-09-17T05:27:00.752 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-17T05:27:00.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-17T05:27:00.754 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-17T05:27:00.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-17T05:27:00.798 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-17T05:27:00.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-17T05:27:00.841 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-17T05:27:00.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-17T05:27:00.885 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-17T05:27:00.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-17T05:27:00.929 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-17T05:27:00.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-17T05:27:00.972 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-17T05:27:00.972 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-17T05:27:01.015 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-17T05:27:01.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-17T05:27:01.059 INFO:tasks.ceph:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-17T05:27:01.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_bal_fragment_size_max 10000 2024-09-17T05:27:01.103 INFO:tasks.ceph:setting config [mds] mds_bal_merge_size = 5 2024-09-17T05:27:01.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_bal_merge_size 5 2024-09-17T05:27:01.154 INFO:tasks.ceph:setting config [mds] mds_bal_split_bits = 3 2024-09-17T05:27:01.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_bal_split_bits 3 2024-09-17T05:27:01.203 INFO:tasks.ceph:setting config [mds] mds_bal_split_size = 100 2024-09-17T05:27:01.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_bal_split_size 100 2024-09-17T05:27:01.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.230+0000 7f7555815640 1 -- 172.21.15.117:0/363174168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7528005290 con 0x7f75500c4d20 2024-09-17T05:27:01.235 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-17T05:27:01.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-17T05:27:01.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.233+0000 7f7537fff640 1 -- 172.21.15.117:0/363174168 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f754804d040 con 0x7f75500c4d20 2024-09-17T05:27:01.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.234+0000 7f7cf53f1640 1 -- 172.21.15.117:0/4069768516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd8005270 msgr2=0x7f7cf01536c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.234+0000 7f7cf53f1640 1 --2- 172.21.15.117:0/4069768516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd8005270 0x7f7cf01536c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006f30 tx=0x7f7ce00039c0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.234+0000 7f7cf53f1640 1 -- 172.21.15.117:0/4069768516 shutdown_connections 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.234+0000 7f7cf53f1640 1 --2- 172.21.15.117:0/4069768516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7cf01ed0c0 0x7f7cf015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.234+0000 7f7cf53f1640 1 --2- 172.21.15.117:0/4069768516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7cf0154600 0x7f7cf015a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.234+0000 7f7cf53f1640 1 --2- 172.21.15.117:0/4069768516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd8005270 0x7f7cf01536c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.235+0000 7f7cf53f1640 1 -- 172.21.15.117:0/4069768516 >> 172.21.15.117:0/4069768516 conn(0x7f7cf006e590 msgr2=0x7f7cf01483e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.235+0000 7f7cf53f1640 1 -- 172.21.15.117:0/4069768516 shutdown_connections 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.235+0000 7f7cf53f1640 1 -- 172.21.15.117:0/4069768516 wait complete. 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.237+0000 7f3a9b7fe640 1 -- 172.21.15.117:0/3487954466 <== mon.2 v2:172.21.15.119:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f3a9c042680 con 0x7f3aa40c5ad0 2024-09-17T05:27:01.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.237+0000 7f7537fff640 1 -- 172.21.15.117:0/363174168 <== mon.1 v2:172.21.15.119:3300/0 8 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f754803fe50 con 0x7f75500c4d20 2024-09-17T05:27:01.244 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-17T05:27:01.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-17T05:27:01.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.239+0000 7fb3b3fff640 1 -- 172.21.15.117:0/167986278 <== mon.0 v2:172.21.15.117:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fb3c400ae00 con 0x7fb3cc150450 2024-09-17T05:27:01.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.241+0000 7effd57fa640 1 -- 172.21.15.117:0/1386010749 <== mon.0 v2:172.21.15.117:3300/0 9 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7effd0025dc0 con 0x7effd8150790 2024-09-17T05:27:01.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.241+0000 7effd57fa640 1 -- 172.21.15.117:0/1386010749 <== mon.0 v2:172.21.15.117:3300/0 10 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7effd0005020 con 0x7effd8150790 2024-09-17T05:27:01.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.247+0000 7f7537fff640 1 -- 172.21.15.117:0/363174168 <== mon.1 v2:172.21.15.119:3300/0 9 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f7548009880 con 0x7f75500c4d20 2024-09-17T05:27:01.254 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-17T05:27:01.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-17T05:27:01.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.249+0000 7f3a9b7fe640 1 -- 172.21.15.117:0/3487954466 <== mon.2 v2:172.21.15.119:3301/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f3a9c0428b0 con 0x7f3aa40c5ad0 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.249+0000 7f3a9b7fe640 1 -- 172.21.15.117:0/3487954466 <== mon.2 v2:172.21.15.119:3301/0 11 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7f3aa40c7960 con 0x7f3aa40c5ad0 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.252+0000 7f3a997fa640 1 -- 172.21.15.117:0/3487954466 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3aa40c5ad0 msgr2=0x7f3aa41f3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 --2- 172.21.15.117:0/3487954466 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3aa40c5ad0 0x7f3aa41f3fc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02aa00 tx=0x7f3a9c004a40 comp rx=0 tx=0).stop 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 -- 172.21.15.117:0/3487954466 shutdown_connections 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 --2- 172.21.15.117:0/3487954466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3aa415ef80 0x7f3aa4157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 --2- 172.21.15.117:0/3487954466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3aa40c61d0 0x7f3aa41e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 --2- 172.21.15.117:0/3487954466 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3aa40c5ad0 0x7f3aa41f3fc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 -- 172.21.15.117:0/3487954466 >> 172.21.15.117:0/3487954466 conn(0x7f3aa406e020 msgr2=0x7f3aa40ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 -- 172.21.15.117:0/3487954466 shutdown_connections 2024-09-17T05:27:01.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7f3a997fa640 1 -- 172.21.15.117:0/3487954466 wait complete. 2024-09-17T05:27:01.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7effded13640 1 -- 172.21.15.117:0/1386010749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effd8150790 msgr2=0x7effd8159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7effd57fa640 1 -- 172.21.15.117:0/1386010749 <== mon.0 v2:172.21.15.117:3300/0 11 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7effd00052b0 con 0x7effd8150790 2024-09-17T05:27:01.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7fb3b3fff640 1 -- 172.21.15.117:0/167986278 <== mon.0 v2:172.21.15.117:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fb3c4010480 con 0x7fb3cc150450 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.253+0000 7effded13640 1 --2- 172.21.15.117:0/1386010749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effd8150790 0x7effd8159820 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7effd80c4730 tx=0x7effd000fcd0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 -- 172.21.15.117:0/1386010749 shutdown_connections 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 --2- 172.21.15.117:0/1386010749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effd8150790 0x7effd8159820 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 --2- 172.21.15.117:0/1386010749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effd8150380 0x7effd81592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 --2- 172.21.15.117:0/1386010749 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7effd8150be0 0x7effd81567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 -- 172.21.15.117:0/1386010749 >> 172.21.15.117:0/1386010749 conn(0x7effd80c9a20 msgr2=0x7effd81589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 -- 172.21.15.117:0/1386010749 shutdown_connections 2024-09-17T05:27:01.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.254+0000 7effded13640 1 -- 172.21.15.117:0/1386010749 wait complete. 2024-09-17T05:27:01.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.256+0000 7f04f8ff9640 1 -- 172.21.15.117:0/4057312106 <== mon.0 v2:172.21.15.117:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f04ec004be0 con 0x7f04fc06deb0 2024-09-17T05:27:01.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.256+0000 7f04f8ff9640 1 -- 172.21.15.117:0/4057312106 <== mon.0 v2:172.21.15.117:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f04ec0106f0 con 0x7f04fc06deb0 2024-09-17T05:27:01.268 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-17T05:27:01.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-17T05:27:01.274 INFO:tasks.ceph:setting config [mds] mds_dir_prefetch = True 2024-09-17T05:27:01.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds mds_dir_prefetch True 2024-09-17T05:27:01.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.295+0000 7fe1089ac640 1 -- 172.21.15.117:0/437617774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2880 msgr2=0x7fe1040cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.301 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-17T05:27:01.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-17T05:27:01.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.295+0000 7fe1089ac640 1 --2- 172.21.15.117:0/437617774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2880 0x7fe1040cbd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe1040c7880 tx=0x7fe0f8032d50 comp rx=0 tx=0).stop 2024-09-17T05:27:01.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.300+0000 7fe1089ac640 1 -- 172.21.15.117:0/437617774 shutdown_connections 2024-09-17T05:27:01.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.301+0000 7fe1089ac640 1 --2- 172.21.15.117:0/437617774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2880 0x7fe1040cbd80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.301+0000 7fe1089ac640 1 --2- 172.21.15.117:0/437617774 >> v2:172.21.15.117:3300/0 conn(0x7fe104155540 0x7fe10415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.301+0000 7fe1089ac640 1 -- 172.21.15.117:0/437617774 >> 172.21.15.117:0/437617774 conn(0x7fe1040c9780 msgr2=0x7fe1040c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.302+0000 7fe1089ac640 1 -- 172.21.15.117:0/437617774 shutdown_connections 2024-09-17T05:27:01.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.302+0000 7fe1089ac640 1 -- 172.21.15.117:0/437617774 wait complete. 2024-09-17T05:27:01.308 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-17T05:27:01.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-17T05:27:01.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.306+0000 7fe1089ac640 1 Processor -- start 2024-09-17T05:27:01.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.306+0000 7fe1089ac640 1 -- start start 2024-09-17T05:27:01.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.308+0000 7fe1089ac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 0x7fe104157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.311 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-17T05:27:01.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-17T05:27:01.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.313+0000 7fe102d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 0x7fe104157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.313+0000 7fe102d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 0x7fe104157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38836/0 (socket says 172.21.15.117:38836) 2024-09-17T05:27:01.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.313+0000 7fe102d9d640 1 -- 172.21.15.117:0/1166231219 learned_addr learned my addr 172.21.15.117:0/1166231219 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.316 INFO:tasks.ceph:setting config [mgr] client_mount_timeout = 30 2024-09-17T05:27:01.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mgr client_mount_timeout 30 2024-09-17T05:27:01.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.314+0000 7fe1089ac640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2c90 0x7fe1040c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.314+0000 7fe1089ac640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe104155540 0x7fe1041fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.314+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe1040c7ea0 con 0x7fe1040c2880 2024-09-17T05:27:01.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.314+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe1040cb3c0 con 0x7fe1040c2c90 2024-09-17T05:27:01.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.314+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe1040c8180 con 0x7fe104155540 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.314+0000 7fe10359e640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe104155540 0x7fe1041fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe10359e640 1 -- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2c90 msgr2=0x7fe1040c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe10359e640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2c90 0x7fe1040c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe10359e640 1 -- 172.21.15.117:0/1166231219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 msgr2=0x7fe104157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe10359e640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 0x7fe104157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe10359e640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1040c4220 con 0x7fe104155540 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe102d9d640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 0x7fe104157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:01.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.315+0000 7fe10359e640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe104155540 0x7fe1041fe9a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00c730 tx=0x7fe0fc00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.317+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc003b70 con 0x7fe104155540 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.317+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fe0fc003d10 con 0x7fe104155540 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.317+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1040c51f0 con 0x7fe104155540 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.317+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe104157920 con 0x7fe104155540 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.318+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc005330 con 0x7fe104155540 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.318+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe0fc005510 con 0x7fe104155540 2024-09-17T05:27:01.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.318+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe0fc012a40 con 0x7fe104155540 2024-09-17T05:27:01.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.320+0000 7fe0e97fa640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e4005290 con 0x7fe104155540 2024-09-17T05:27:01.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.324+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe0fc00a640 con 0x7fe104155540 2024-09-17T05:27:01.331 INFO:tasks.ceph:setting config [mgr] debug_client = 20 2024-09-17T05:27:01.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mgr debug_client 20 2024-09-17T05:27:01.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.361+0000 7f3a2324d640 1 -- 172.21.15.117:0/528666704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c4d20 msgr2=0x7f3a1c0c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.361+0000 7f3a2324d640 1 --2- 172.21.15.117:0/528666704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c0c4480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0c8fb0 tx=0x7f3a10032b10 comp rx=0 tx=0).stop 2024-09-17T05:27:01.363 INFO:tasks.ceph:setting config [mgr] debug_mgr = 20 2024-09-17T05:27:01.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mgr debug_mgr 20 2024-09-17T05:27:01.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.364+0000 7f3a2324d640 1 -- 172.21.15.117:0/528666704 shutdown_connections 2024-09-17T05:27:01.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.364+0000 7f3a2324d640 1 --2- 172.21.15.117:0/528666704 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c0c85d0 0x7f3a1c074e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.364+0000 7f3a2324d640 1 --2- 172.21.15.117:0/528666704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c0c4480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.364+0000 7f3a2324d640 1 -- 172.21.15.117:0/528666704 >> 172.21.15.117:0/528666704 conn(0x7f3a1c0ccbe0 msgr2=0x7f3a1c0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.367 INFO:tasks.ceph:setting config [mgr] debug_ms = 1 2024-09-17T05:27:01.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mgr debug_ms 1 2024-09-17T05:27:01.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.364+0000 7f3a2324d640 1 -- 172.21.15.117:0/528666704 shutdown_connections 2024-09-17T05:27:01.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.364+0000 7f3a2324d640 1 -- 172.21.15.117:0/528666704 wait complete. 2024-09-17T05:27:01.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.367+0000 7f3a2324d640 1 Processor -- start 2024-09-17T05:27:01.369 INFO:tasks.ceph:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-17T05:27:01.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mgr mon_warn_on_pool_no_app False 2024-09-17T05:27:01.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.367+0000 7f3a2324d640 1 -- start start 2024-09-17T05:27:01.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.369+0000 7f3a2324d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 0x7f3a1c1674a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.370+0000 7f3a2324d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c165e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.374 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-17T05:27:01.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-17T05:27:01.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.370+0000 7f3a2324d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c85d0 0x7f3a1c086060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.370+0000 7f3a20fe9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 0x7f3a1c1674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.370+0000 7f3a20fe9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 0x7f3a1c1674a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:46334/0 (socket says 172.21.15.117:46334) 2024-09-17T05:27:01.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.370+0000 7f3a1bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c165e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.370+0000 7f3a1bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c165e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:48864/0 (socket says 172.21.15.117:48864) 2024-09-17T05:27:01.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.391+0000 7f3a217ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c85d0 0x7f3a1c086060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.391+0000 7f3a217ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c85d0 0x7f3a1c086060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38850/0 (socket says 172.21.15.117:38850) 2024-09-17T05:27:01.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.391+0000 7f3a2324d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c0c7660 con 0x7f3a1c0c85d0 2024-09-17T05:27:01.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.391+0000 7f3a2324d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c08cd10 con 0x7f3a1c0c4d20 2024-09-17T05:27:01.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.391+0000 7f3a2324d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1c167eb0 con 0x7f3a1c072e60 2024-09-17T05:27:01.393 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-17T05:27:01.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-17T05:27:01.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.392+0000 7f3a20fe9640 1 -- 172.21.15.117:0/159353207 learned_addr learned my addr 172.21.15.117:0/159353207 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.392+0000 7f3a20fe9640 1 -- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a1c0c4d20 msgr2=0x7f3a1c165e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.392+0000 7f3a20fe9640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c165e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.392+0000 7f3a20fe9640 1 -- 172.21.15.117:0/159353207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c85d0 msgr2=0x7f3a1c086060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.392+0000 7f3a20fe9640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c85d0 0x7f3a1c086060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.392+0000 7f3a20fe9640 1 -- 172.21.15.117:0/159353207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c1679e0 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.395+0000 7f050252b640 1 -- 172.21.15.117:0/4057312106 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f04d40052f0 con 0x7f04fc06deb0 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.393+0000 7f3a20fe9640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 0x7f3a1c1674a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3a10033100 tx=0x7f3a10003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.395+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10042070 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.395+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f3a10012370 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.395+0000 7f3a2324d640 1 -- 172.21.15.117:0/159353207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c16a910 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.395+0000 7f3a2324d640 1 -- 172.21.15.117:0/159353207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c166ca0 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.396+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10012780 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.397+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a10045070 con 0x7f3a1c072e60 2024-09-17T05:27:01.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.397+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3a10034960 con 0x7f3a1c072e60 2024-09-17T05:27:01.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.400+0000 7f7537fff640 1 -- 172.21.15.117:0/363174168 <== mon.1 v2:172.21.15.119:3300/0 10 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f754803ddb0 con 0x7f75500c4d20 2024-09-17T05:27:01.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.399+0000 7f3a2324d640 1 -- 172.21.15.117:0/159353207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39fc005290 con 0x7f3a1c072e60 2024-09-17T05:27:01.411 DEBUG:tasks.ceph:set 27 configs 2024-09-17T05:27:01.411 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.412 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.403+0000 7f04f8ff9640 1 -- 172.21.15.117:0/4057312106 <== mon.0 v2:172.21.15.117:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v6)=0 v6) ==== 114+0+0 (secure 0 0 0) 0x7f04ec010a10 con 0x7f04fc06deb0 2024-09-17T05:27:01.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.402+0000 7fb3b3fff640 1 -- 172.21.15.117:0/167986278 <== mon.0 v2:172.21.15.117:3300/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3c40106d0 con 0x7fb3cc150450 2024-09-17T05:27:01.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.401+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0fc005800 con 0x7fe104155540 2024-09-17T05:27:01.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.403+0000 7f04f8ff9640 1 -- 172.21.15.117:0/4057312106 <== mon.0 v2:172.21.15.117:3300/0 12 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04ec010d60 con 0x7f04fc06deb0 2024-09-17T05:27:01.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 -- 172.21.15.117:0/4057312106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc06deb0 msgr2=0x7f04fc1599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 --2- 172.21.15.117:0/4057312106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc06deb0 0x7f04fc1599c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00d720 tx=0x7f04ec00dbe0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 -- 172.21.15.117:0/4057312106 shutdown_connections 2024-09-17T05:27:01.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 --2- 172.21.15.117:0/4057312106 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f04fc0c47b0 0x7f04fc0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 --2- 172.21.15.117:0/4057312106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc0c43a0 0x7f04fc14dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 --2- 172.21.15.117:0/4057312106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc06deb0 0x7f04fc1599c0 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00d720 tx=0x7f04ec00dbe0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.416+0000 7f04e27fc640 1 -- 172.21.15.117:0/4057312106 >> 172.21.15.117:0/4057312106 conn(0x7f04fc0c9a20 msgr2=0x7f04fc0c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.418+0000 7f04e27fc640 1 -- 172.21.15.117:0/4057312106 shutdown_connections 2024-09-17T05:27:01.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.418+0000 7f04e27fc640 1 -- 172.21.15.117:0/4057312106 wait complete. 2024-09-17T05:27:01.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.418+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 6 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a10004030 con 0x7f3a1c072e60 2024-09-17T05:27:01.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.418+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f3a1004c020 con 0x7f3a1c072e60 2024-09-17T05:27:01.469 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.469 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.469 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.469 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.471+0000 7fb3d1943640 1 -- 172.21.15.117:0/167986278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fb3a40052f0 con 0x7fb3cc150450 2024-09-17T05:27:01.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.478+0000 7fb3b3fff640 1 -- 172.21.15.117:0/167986278 <== mon.0 v2:172.21.15.117:3300/0 12 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v7)=0 v7) ==== 108+0+0 (secure 0 0 0) 0x7fb3c4010a40 con 0x7fb3cc150450 2024-09-17T05:27:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.480+0000 7fb3d1943640 1 -- 172.21.15.117:0/167986278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3cc150450 msgr2=0x7fb3cc154a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.481+0000 7fb3d1943640 1 --2- 172.21.15.117:0/167986278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3cc150450 0x7fb3cc154a00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400c880 tx=0x7fb3c400cd40 comp rx=0 tx=0).stop 2024-09-17T05:27:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.481+0000 7fb3d1943640 1 -- 172.21.15.117:0/167986278 shutdown_connections 2024-09-17T05:27:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.482+0000 7fb3d1943640 1 --2- 172.21.15.117:0/167986278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3cc150450 0x7fb3cc154a00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.482+0000 7fb3d1943640 1 --2- 172.21.15.117:0/167986278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3cc0caba0 0x7fb3cc156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.482+0000 7fb3d1943640 1 --2- 172.21.15.117:0/167986278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3cc0c8500 0x7fb3cc0cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.482+0000 7fb3d1943640 1 -- 172.21.15.117:0/167986278 >> 172.21.15.117:0/167986278 conn(0x7fb3cc0c9a20 msgr2=0x7fb3cc0bfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.484+0000 7fb3d1943640 1 -- 172.21.15.117:0/167986278 shutdown_connections 2024-09-17T05:27:01.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.485+0000 7fb3d1943640 1 -- 172.21.15.117:0/167986278 wait complete. 2024-09-17T05:27:01.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.524+0000 7f80c218b640 1 -- 172.21.15.117:0/2771553231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 msgr2=0x7f80bc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.524+0000 7f80c218b640 1 --2- 172.21.15.117:0/2771553231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 0x7f80bc06cda0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0749e0 tx=0x7f80b402a840 comp rx=0 tx=0).stop 2024-09-17T05:27:01.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.525+0000 7f80c218b640 1 -- 172.21.15.117:0/2771553231 shutdown_connections 2024-09-17T05:27:01.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.525+0000 7f80c218b640 1 --2- 172.21.15.117:0/2771553231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 0x7f80bc06cda0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.525+0000 7f80c218b640 1 --2- 172.21.15.117:0/2771553231 >> v2:172.21.15.117:3300/0 conn(0x7f80bc15b220 0x7f80bc0c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.525+0000 7f80c218b640 1 -- 172.21.15.117:0/2771553231 >> 172.21.15.117:0/2771553231 conn(0x7f80bc0ccbe0 msgr2=0x7f80bc0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.525+0000 7f80c218b640 1 -- 172.21.15.117:0/2771553231 shutdown_connections 2024-09-17T05:27:01.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.525+0000 7f80c218b640 1 -- 172.21.15.117:0/2771553231 wait complete. 2024-09-17T05:27:01.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.528+0000 7f80c218b640 1 Processor -- start 2024-09-17T05:27:01.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.533+0000 7f80c218b640 1 -- start start 2024-09-17T05:27:01.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.536+0000 7f80c218b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 0x7f80bc1529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.536+0000 7f80bb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 0x7f80bc1529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.536+0000 7f80bb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 0x7f80bc1529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38856/0 (socket says 172.21.15.117:38856) 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.536+0000 7f80bb7fe640 1 -- 172.21.15.117:0/3391369988 learned_addr learned my addr 172.21.15.117:0/3391369988 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.537+0000 7f80c218b640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 0x7f80bc150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.537+0000 7f80c218b640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc15b220 0x7f80bc148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.537+0000 7f80c218b640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc0744f0 con 0x7f80bc074cf0 2024-09-17T05:27:01.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.537+0000 7f80bbfff640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc15b220 0x7f80bc148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.538+0000 7f80c218b640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc074210 con 0x7f80bc15b220 2024-09-17T05:27:01.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.538+0000 7f80c218b640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f80bc0733b0 con 0x7f80bc1551b0 2024-09-17T05:27:01.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.539+0000 7f80bb7fe640 1 -- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 msgr2=0x7f80bc150820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.539+0000 7f80bb7fe640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 0x7f80bc150820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.539+0000 7f80bb7fe640 1 -- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc15b220 msgr2=0x7f80bc148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.539+0000 7f80bb7fe640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc15b220 0x7f80bc148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.539+0000 7f80bb7fe640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc150d60 con 0x7f80bc074cf0 2024-09-17T05:27:01.551 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.554+0000 7f80bb7fe640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 0x7f80bc1529a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00ed60 tx=0x7f80ac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.555+0000 7f80bbfff640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc15b220 0x7f80bc148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:01.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.555+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00ee50 con 0x7f80bc074cf0 2024-09-17T05:27:01.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.555+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80ac004020 con 0x7f80bc074cf0 2024-09-17T05:27:01.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.555+0000 7f80c218b640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80bc1532e0 con 0x7f80bc074cf0 2024-09-17T05:27:01.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.556+0000 7f80c218b640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80bc1f1a50 con 0x7f80bc074cf0 2024-09-17T05:27:01.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.557+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac004450 con 0x7f80bc074cf0 2024-09-17T05:27:01.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.559+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f80ac003980 con 0x7f80bc074cf0 2024-09-17T05:27:01.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.559+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f80ac010770 con 0x7f80bc074cf0 2024-09-17T05:27:01.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.559+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809c005290 con 0x7f80bc074cf0 2024-09-17T05:27:01.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.565+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f80ac026070 con 0x7f80bc074cf0 2024-09-17T05:27:01.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.698+0000 7fed0d293640 1 -- 172.21.15.117:0/1038604044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed08074cd0 msgr2=0x7fed080c4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.698+0000 7fed0d293640 1 --2- 172.21.15.117:0/1038604044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed08074cd0 0x7fed080c4420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fed080cd440 tx=0x7fecf002a620 comp rx=0 tx=0).stop 2024-09-17T05:27:01.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.702+0000 7fed0d293640 1 -- 172.21.15.117:0/1038604044 shutdown_connections 2024-09-17T05:27:01.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.702+0000 7fed0d293640 1 --2- 172.21.15.117:0/1038604044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed08074cd0 0x7fed080c4420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.702+0000 7fed0d293640 1 --2- 172.21.15.117:0/1038604044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0806cf80 0x7fed0815e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.702+0000 7fed0d293640 1 --2- 172.21.15.117:0/1038604044 >> v2:172.21.15.119:3300/0 conn(0x7fed0815b220 0x7fed080c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.702+0000 7fed0d293640 1 -- 172.21.15.117:0/1038604044 >> 172.21.15.117:0/1038604044 conn(0x7fed080ccbe0 msgr2=0x7fed080c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.707+0000 7f7555815640 1 -- 172.21.15.117:0/363174168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f7528005850 con 0x7f75500c4d20 2024-09-17T05:27:01.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.708+0000 7fe0e97fa640 1 -- 172.21.15.117:0/1166231219 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fe0e4005580 con 0x7fe104155540 2024-09-17T05:27:01.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.710+0000 7fed0d293640 1 -- 172.21.15.117:0/1038604044 shutdown_connections 2024-09-17T05:27:01.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.710+0000 7f63239df640 1 -- 172.21.15.117:0/234421946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f631404d260 msgr2=0x7f63140def50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.710+0000 7f63239df640 1 --2- 172.21.15.117:0/234421946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f631404d260 0x7f63140def50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6318005fd0 tx=0x7f631802a5e0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.711+0000 7f63239df640 1 -- 172.21.15.117:0/234421946 shutdown_connections 2024-09-17T05:27:01.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.711+0000 7f63239df640 1 --2- 172.21.15.117:0/234421946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f63140df490 0x7f63140e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.711+0000 7f63239df640 1 --2- 172.21.15.117:0/234421946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f631404d260 0x7f63140def50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.711+0000 7f63239df640 1 -- 172.21.15.117:0/234421946 >> 172.21.15.117:0/234421946 conn(0x7f6314050f50 msgr2=0x7f6314051360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.710+0000 7fed0d293640 1 -- 172.21.15.117:0/1038604044 wait complete. 2024-09-17T05:27:01.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.713+0000 7fed0d293640 1 Processor -- start 2024-09-17T05:27:01.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 -- 172.21.15.117:0/1691393212 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 msgr2=0x7f48ec005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 --2- 172.21.15.117:0/1691393212 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 0x7f48ec005c00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48f40524b0 tx=0x7f48f40afd10 comp rx=0 tx=0).stop 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.711+0000 7f63239df640 1 -- 172.21.15.117:0/234421946 shutdown_connections 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.713+0000 7fed0d293640 1 -- start start 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed0d293640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed0806cf80 0x7fed0815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed0d293640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed08074cd0 0x7fed080c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed0d293640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 0x7fed08153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed0d293640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed080744d0 con 0x7fed0806cf80 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed0d293640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fed080741f0 con 0x7fed08074cd0 2024-09-17T05:27:01.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed0d293640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fed080733b0 con 0x7fed0815b220 2024-09-17T05:27:01.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed077fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 0x7fed08153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed077fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 0x7fed08153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:46392/0 (socket says 172.21.15.117:46392) 2024-09-17T05:27:01.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7fed077fe640 1 -- 172.21.15.117:0/2934591313 learned_addr learned my addr 172.21.15.117:0/2934591313 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.716+0000 7fecfffff640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed08074cd0 0x7fed080c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.715+0000 7f63239df640 1 -- 172.21.15.117:0/234421946 wait complete. 2024-09-17T05:27:01.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.716+0000 7fed077fe640 1 -- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed08074cd0 msgr2=0x7fed080c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.718+0000 7f63239df640 1 Processor -- start 2024-09-17T05:27:01.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.718+0000 7f63239df640 1 -- start start 2024-09-17T05:27:01.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 -- 172.21.15.117:0/1691393212 shutdown_connections 2024-09-17T05:27:01.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 --2- 172.21.15.117:0/1691393212 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 0x7f48ec005c00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 --2- 172.21.15.117:0/1691393212 >> v2:172.21.15.119:3300/0 conn(0x7f48ec044cb0 0x7f48ec0d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 --2- 172.21.15.117:0/1691393212 >> v2:172.21.15.117:3300/0 conn(0x7f48ec00de50 0x7f48ec0db6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.714+0000 7f48faf58640 1 -- 172.21.15.117:0/1691393212 >> 172.21.15.117:0/1691393212 conn(0x7f48ec04fe30 msgr2=0x7f48ec050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f63239df640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f631404d260 0x7f63140de8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f63239df640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63140d2590 0x7f63140c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f63239df640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63140df490 0x7f63140c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f7537fff640 1 -- 172.21.15.117:0/363174168 <== mon.1 v2:172.21.15.119:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7f7548043030 con 0x7f75500c4d20 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f48faf58640 1 -- 172.21.15.117:0/1691393212 shutdown_connections 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f63239df640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f63140df8a0 con 0x7f63140df490 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f63239df640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6314008500 con 0x7f63140d2590 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f63239df640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f63140dfaf0 con 0x7f631404d260 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f6321f7c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63140df490 0x7f63140c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f6321f7c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63140df490 0x7f63140c17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38894/0 (socket says 172.21.15.117:38894) 2024-09-17T05:27:01.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.720+0000 7f6321f7c640 1 -- 172.21.15.117:0/3180130072 learned_addr learned my addr 172.21.15.117:0/3180130072 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f632177b640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f631404d260 0x7f63140de8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f6320f7a640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63140d2590 0x7f63140c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f632177b640 1 -- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63140d2590 msgr2=0x7f63140c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f632177b640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63140d2590 0x7f63140c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f632177b640 1 -- 172.21.15.117:0/3180130072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63140df490 msgr2=0x7f63140c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f632177b640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63140df490 0x7f63140c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.721+0000 7f632177b640 1 -- 172.21.15.117:0/3180130072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63140c7a70 con 0x7f631404d260 2024-09-17T05:27:01.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.716+0000 7fed077fe640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed08074cd0 0x7fed080c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.716+0000 7fed077fe640 1 -- 172.21.15.117:0/2934591313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed0806cf80 msgr2=0x7fed0815dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.716+0000 7fed077fe640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed0806cf80 0x7fed0815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.716+0000 7fed077fe640 1 -- 172.21.15.117:0/2934591313 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed081fea60 con 0x7fed0815b220 2024-09-17T05:27:01.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.722+0000 7fed077fe640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 0x7fed08153f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fed0000b9c0 tx=0x7fed0000be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.722+0000 7f632177b640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f631404d260 0x7f63140de8c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f630c00c3c0 tx=0x7f630c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 -- 172.21.15.117:0/363174168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75500c4d20 msgr2=0x7f755014dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 --2- 172.21.15.117:0/363174168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75500c4d20 0x7f755014dfd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7548009d60 tx=0x7f7548004a40 comp rx=0 tx=0).stop 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 -- 172.21.15.117:0/363174168 shutdown_connections 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 --2- 172.21.15.117:0/363174168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755015b220 0x7f755014e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 --2- 172.21.15.117:0/363174168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75500c4d20 0x7f755014dfd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 --2- 172.21.15.117:0/363174168 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75500c4420 0x7f75501f0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.725+0000 7f7535ffb640 1 -- 172.21.15.117:0/363174168 >> 172.21.15.117:0/363174168 conn(0x7f75500ccbe0 msgr2=0x7f7550068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.726+0000 7fe0eb7fe640 1 -- 172.21.15.117:0/1166231219 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7fe0fc00a820 con 0x7fe104155540 2024-09-17T05:27:01.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.727+0000 7f48faf58640 1 -- 172.21.15.117:0/1691393212 wait complete. 2024-09-17T05:27:01.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.727+0000 7f7535ffb640 1 -- 172.21.15.117:0/363174168 shutdown_connections 2024-09-17T05:27:01.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.728+0000 7f7535ffb640 1 -- 172.21.15.117:0/363174168 wait complete. 2024-09-17T05:27:01.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.730+0000 7f48faf58640 1 Processor -- start 2024-09-17T05:27:01.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.732+0000 7f48faf58640 1 -- start start 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.734+0000 7f48faf58640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec00de50 0x7f48ec161d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.734+0000 7f48faf58640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 0x7f48ec049890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.728+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c007030 con 0x7f631404d260 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.728+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f630c0071d0 con 0x7f631404d260 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.728+0000 7f63239df640 1 -- 172.21.15.117:0/3180130072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63140ddde0 con 0x7f631404d260 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.728+0000 7f63239df640 1 -- 172.21.15.117:0/3180130072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63140de3a0 con 0x7f631404d260 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.728+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c007620 con 0x7f631404d260 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.729+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f630c01a020 con 0x7f631404d260 2024-09-17T05:27:01.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.729+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f630c013400 con 0x7f631404d260 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.732+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631404d040 con 0x7f631404d260 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.732+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00019070 con 0x7fed0815b220 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.732+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed0000cc90 con 0x7fed0815b220 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.732+0000 7fed0d293640 1 -- 172.21.15.117:0/2934591313 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed081fefc0 con 0x7fed0815b220 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.732+0000 7fed0d293640 1 -- 172.21.15.117:0/2934591313 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed08150d30 con 0x7fed0815b220 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.734+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed000146a0 con 0x7fed0815b220 2024-09-17T05:27:01.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.734+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0806d390 con 0x7fed0815b220 2024-09-17T05:27:01.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe104155540 msgr2=0x7fe1041fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe104155540 0x7fe1041fe9a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00c730 tx=0x7fe0fc00cbf0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 0x7f48ec049890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 0x7f48ec049890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38902/0 (socket says 172.21.15.117:38902) 2024-09-17T05:27:01.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 -- 172.21.15.117:0/3484324191 learned_addr learned my addr 172.21.15.117:0/3484324191 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 shutdown_connections 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe104155540 0x7fe1041fe9a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1040c2c90 0x7fe1040c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 --2- 172.21.15.117:0/1166231219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1040c2880 0x7fe104157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 >> 172.21.15.117:0/1166231219 conn(0x7fe1040c9780 msgr2=0x7fe104154290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 shutdown_connections 2024-09-17T05:27:01.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.736+0000 7fe1089ac640 1 -- 172.21.15.117:0/1166231219 wait complete. 2024-09-17T05:27:01.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48faf58640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 0x7f48ec0c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec0461d0 con 0x7f48ec044cb0 2024-09-17T05:27:01.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec045ef0 con 0x7f48ec00de50 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec0d5200 con 0x7f48ec0469d0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 -- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 msgr2=0x7f48ec0c75b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 0x7f48ec0c75b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 -- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec00de50 msgr2=0x7f48ec161d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.735+0000 7f48f3fff640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec00de50 0x7f48ec161d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.736+0000 7f48f3fff640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec1627e0 con 0x7f48ec044cb0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.736+0000 7f48f3fff640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 0x7f48ec049890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48e000ca30 tx=0x7f48e000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.736+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0007b20 con 0x7f48ec044cb0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.737+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ec163510 con 0x7f48ec044cb0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.737+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ec0cb370 con 0x7f48ec044cb0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.738+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48e0007dd0 con 0x7f48ec044cb0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.738+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0005280 con 0x7f48ec044cb0 2024-09-17T05:27:01.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.739+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f48e00047a0 con 0x7f48ec044cb0 2024-09-17T05:27:01.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.741+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed0001c070 con 0x7fed0815b220 2024-09-17T05:27:01.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.742+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f48e000fe00 con 0x7f48ec044cb0 2024-09-17T05:27:01.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.742+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed0001e940 con 0x7fed0815b220 2024-09-17T05:27:01.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.742+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fed00023020 con 0x7fed0815b220 2024-09-17T05:27:01.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.742+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d4005290 con 0x7f48ec044cb0 2024-09-17T05:27:01.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.747+0000 7f52729eb640 1 -- 172.21.15.117:0/3522688642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 msgr2=0x7f5264007950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.747+0000 7f52729eb640 1 --2- 172.21.15.117:0/3522688642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f5264007950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f526400d660 tx=0x7f526802a610 comp rx=0 tx=0).stop 2024-09-17T05:27:01.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.748+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f48e0014030 con 0x7f48ec044cb0 2024-09-17T05:27:01.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.748+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f630c021070 con 0x7f631404d260 2024-09-17T05:27:01.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.760+0000 7f52729eb640 1 -- 172.21.15.117:0/3522688642 shutdown_connections 2024-09-17T05:27:01.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.760+0000 7f52729eb640 1 --2- 172.21.15.117:0/3522688642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f5264007950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.760+0000 7f52729eb640 1 --2- 172.21.15.117:0/3522688642 >> v2:172.21.15.119:3300/0 conn(0x7f526404bbe0 0x7f5264034440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.760+0000 7f52729eb640 1 -- 172.21.15.117:0/3522688642 >> 172.21.15.117:0/3522688642 conn(0x7f5264050710 msgr2=0x7f5264050b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.762+0000 7f52729eb640 1 -- 172.21.15.117:0/3522688642 shutdown_connections 2024-09-17T05:27:01.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.766+0000 7f52729eb640 1 -- 172.21.15.117:0/3522688642 wait complete. 2024-09-17T05:27:01.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.768+0000 7f52729eb640 1 Processor -- start 2024-09-17T05:27:01.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.769+0000 7f52729eb640 1 -- start start 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52729eb640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f526404bbe0 0x7f52640da240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52729eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f52640cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52729eb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52640de6b0 0x7f52640c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52711e8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f52640cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52711e8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f52640cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38926/0 (socket says 172.21.15.117:38926) 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52711e8640 1 -- 172.21.15.117:0/90101550 learned_addr learned my addr 172.21.15.117:0/90101550 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f526404c3c0 con 0x7f52640d5040 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f526404c0e0 con 0x7f52640de6b0 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f52640d5450 con 0x7f526404bbe0 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52721ea640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52640de6b0 0x7f52640c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.771+0000 7f52719e9640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f526404bbe0 0x7f52640da240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52721ea640 1 -- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f526404bbe0 msgr2=0x7f52640da240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52721ea640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f526404bbe0 0x7f52640da240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52721ea640 1 -- 172.21.15.117:0/90101550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 msgr2=0x7f52640cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52721ea640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f52640cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52721ea640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52640d39f0 con 0x7f52640de6b0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52721ea640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52640de6b0 0x7f52640c9410 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f525c00b410 tx=0x7f525c00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c015020 con 0x7f52640de6b0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.772+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f525c011830 con 0x7f52640de6b0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.773+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52640d2f50 con 0x7f52640de6b0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.773+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52640d31f0 con 0x7f52640de6b0 2024-09-17T05:27:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.773+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c011c80 con 0x7f52640de6b0 2024-09-17T05:27:01.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.779+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5254005290 con 0x7f52640de6b0 2024-09-17T05:27:01.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.781+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f525c018020 con 0x7f52640de6b0 2024-09-17T05:27:01.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.781+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f525c019ab0 con 0x7f52640de6b0 2024-09-17T05:27:01.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.784+0000 7f76ddbe6640 1 -- 172.21.15.117:0/3699818398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c0003ad0 msgr2=0x7f76c0003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.784+0000 7fe9c5427640 1 -- 172.21.15.117:0/507750528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8044d50 msgr2=0x7fe9b80454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.784+0000 7fe9c5427640 1 --2- 172.21.15.117:0/507750528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8044d50 0x7fe9b80454e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800dc80 tx=0x7fe9b402a530 comp rx=0 tx=0).stop 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.787+0000 7fe9c5427640 1 -- 172.21.15.117:0/507750528 shutdown_connections 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.787+0000 7fe9c5427640 1 --2- 172.21.15.117:0/507750528 >> v2:172.21.15.119:3300/0 conn(0x7fe9b8005c30 0x7fe9b8006040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.787+0000 7fe9c5427640 1 --2- 172.21.15.117:0/507750528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8044d50 0x7fe9b80454e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.787+0000 7fe9c5427640 1 -- 172.21.15.117:0/507750528 >> 172.21.15.117:0/507750528 conn(0x7fe9b804fe60 msgr2=0x7fe9b8050270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.787+0000 7fe9c5427640 1 -- 172.21.15.117:0/507750528 shutdown_connections 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.787+0000 7fe9c5427640 1 -- 172.21.15.117:0/507750528 wait complete. 2024-09-17T05:27:01.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.790+0000 7fe9c5427640 1 Processor -- start 2024-09-17T05:27:01.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.790+0000 7fe9c5427640 1 -- start start 2024-09-17T05:27:01.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.792+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f525c01f020 con 0x7f52640de6b0 2024-09-17T05:27:01.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.792+0000 7fe9c5427640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8005c30 0x7fe9b80d5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.784+0000 7f76ddbe6640 1 --2- 172.21.15.117:0/3699818398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c0003ad0 0x7f76c0003f00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76c800bb50 tx=0x7f76c0005830 comp rx=0 tx=0).stop 2024-09-17T05:27:01.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.792+0000 7fe9c5427640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 0x7fe9b804e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.792+0000 7f76d7fff640 1 -- 172.21.15.117:0/3699818398 reap_dead start 2024-09-17T05:27:01.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9beffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8005c30 0x7fe9b80d5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7f76ddbe6640 1 -- 172.21.15.117:0/3699818398 shutdown_connections 2024-09-17T05:27:01.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7f76ddbe6640 1 -- 172.21.15.117:0/3699818398 >> 172.21.15.117:0/3699818398 conn(0x7f76d80ccb40 msgr2=0x7f76d80ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.795 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.795 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9be7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 0x7fe9b804e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9be7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 0x7fe9b804e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:48970/0 (socket says 172.21.15.117:48970) 2024-09-17T05:27:01.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9c5427640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9b80df650 0x7fe9b80cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9be7fc640 1 -- 172.21.15.117:0/3705834645 learned_addr learned my addr 172.21.15.117:0/3705834645 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9bf7fe640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9b80df650 0x7fe9b80cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7f76ddbe6640 1 -- 172.21.15.117:0/3699818398 shutdown_connections 2024-09-17T05:27:01.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9c5427640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b8046600 con 0x7fe9b8005c30 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9c5427640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b8006580 con 0x7fe9b8044d50 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7fe9c5427640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b80468e0 con 0x7fe9b80df650 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.794+0000 7fe9be7fc640 1 -- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9b80df650 msgr2=0x7fe9b80cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.794+0000 7fe9be7fc640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9b80df650 0x7fe9b80cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.794+0000 7fe9be7fc640 1 -- 172.21.15.117:0/3705834645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8005c30 msgr2=0x7fe9b80d5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.794+0000 7fe9be7fc640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8005c30 0x7fe9b80d5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.794+0000 7fe9be7fc640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b80d5750 con 0x7fe9b8044d50 2024-09-17T05:27:01.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.793+0000 7f76ddbe6640 1 -- 172.21.15.117:0/3699818398 wait complete. 2024-09-17T05:27:01.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.795+0000 7fe9beffd640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8005c30 0x7fe9b80d5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:01.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.795+0000 7fe9be7fc640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 0x7fe9b804e260 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000c520 tx=0x7fe9b000c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.795+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0017070 con 0x7fe9b8044d50 2024-09-17T05:27:01.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.796+0000 7f76ddbe6640 1 Processor -- start 2024-09-17T05:27:01.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.795+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9b0005480 con 0x7fe9b8044d50 2024-09-17T05:27:01.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.795+0000 7fe9c5427640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b817a570 con 0x7fe9b8044d50 2024-09-17T05:27:01.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.795+0000 7fe9c5427640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b80d6a10 con 0x7fe9b8044d50 2024-09-17T05:27:01.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.797+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0003850 con 0x7fe9b8044d50 2024-09-17T05:27:01.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.800+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe9b001a070 con 0x7fe9b8044d50 2024-09-17T05:27:01.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.796+0000 7f76ddbe6640 1 -- start start 2024-09-17T05:27:01.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.804+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe9b001c780 con 0x7fe9b8044d50 2024-09-17T05:27:01.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.804+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a0005290 con 0x7fe9b8044d50 2024-09-17T05:27:01.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.805+0000 7f76ddbe6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 0x7f76d80c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.805+0000 7f76ddbe6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76d81f8b10 0x7f76d81f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.805+0000 7f76ddbe6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f76d814f390 0x7f76d814f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.805+0000 7f76ddbe6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f76d80cc300 con 0x7f76d815e4b0 2024-09-17T05:27:01.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.805+0000 7f76ddbe6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f76d80cb2e0 con 0x7f76d81f8b10 2024-09-17T05:27:01.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.805+0000 7f76ddbe6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f76d806d7f0 con 0x7f76d814f390 2024-09-17T05:27:01.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.808+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 0x7f76d80c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.808+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 0x7f76d80c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38946/0 (socket says 172.21.15.117:38946) 2024-09-17T05:27:01.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.808+0000 7f76d77fe640 1 -- 172.21.15.117:0/2995019052 learned_addr learned my addr 172.21.15.117:0/2995019052 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d6ffd640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76d81f8b10 0x7f76d81f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d7fff640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f76d814f390 0x7f76d814f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe9b00039f0 con 0x7fe9b8044d50 2024-09-17T05:27:01.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d77fe640 1 -- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f76d814f390 msgr2=0x7f76d814f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d77fe640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f76d814f390 0x7f76d814f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d77fe640 1 -- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76d81f8b10 msgr2=0x7f76d81f8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d77fe640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76d81f8b10 0x7f76d81f8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.809+0000 7f76d77fe640 1 -- 172.21.15.117:0/2995019052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d81584d0 con 0x7f76d815e4b0 2024-09-17T05:27:01.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.810+0000 7f76d77fe640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 0x7f76d80c2950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f76c800baf0 tx=0x7f76c802f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.810+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c802f860 con 0x7f76d815e4b0 2024-09-17T05:27:01.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.811+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76c802fa00 con 0x7f76d815e4b0 2024-09-17T05:27:01.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.811+0000 7f76ddbe6640 1 -- 172.21.15.117:0/2995019052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d81f9630 con 0x7f76d815e4b0 2024-09-17T05:27:01.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.811+0000 7f76ddbe6640 1 -- 172.21.15.117:0/2995019052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d80c7a40 con 0x7f76d815e4b0 2024-09-17T05:27:01.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.813+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c802fe50 con 0x7f76d815e4b0 2024-09-17T05:27:01.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.825+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f76c8038020 con 0x7f76d815e4b0 2024-09-17T05:27:01.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.826+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f76c804d5c0 con 0x7f76d815e4b0 2024-09-17T05:27:01.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.827+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b8005290 con 0x7f76d815e4b0 2024-09-17T05:27:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa01764d640 1 -- 172.21.15.117:0/2861615710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ffc004800 msgr2=0x7f9ffc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa01764d640 1 --2- 172.21.15.117:0/2861615710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ffc004800 0x7f9ffc004c10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa00c006f20 tx=0x7f9ffc005830 comp rx=0 tx=0).stop 2024-09-17T05:27:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa015bea640 1 -- 172.21.15.117:0/2861615710 reap_dead start 2024-09-17T05:27:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa01764d640 1 -- 172.21.15.117:0/2861615710 shutdown_connections 2024-09-17T05:27:01.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa01764d640 1 -- 172.21.15.117:0/2861615710 >> 172.21.15.117:0/2861615710 conn(0x7fa0100cc940 msgr2=0x7fa0100c8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa01764d640 1 -- 172.21.15.117:0/2861615710 shutdown_connections 2024-09-17T05:27:01.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.831+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f76c803f070 con 0x7f76d815e4b0 2024-09-17T05:27:01.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.830+0000 7fa01764d640 1 -- 172.21.15.117:0/2861615710 wait complete. 2024-09-17T05:27:01.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.833+0000 7fa01764d640 1 Processor -- start 2024-09-17T05:27:01.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.834+0000 7fa63fb41640 1 -- 172.21.15.117:0/251549797 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6380c9940 msgr2=0x7fa6381680a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.834+0000 7fa63fb41640 1 --2- 172.21.15.117:0/251549797 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6380c9940 0x7fa6381680a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa6340097b0 tx=0x7fa63402a420 comp rx=0 tx=0).stop 2024-09-17T05:27:01.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.836+0000 7fa63fb41640 1 -- 172.21.15.117:0/251549797 shutdown_connections 2024-09-17T05:27:01.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.836+0000 7fa63fb41640 1 --2- 172.21.15.117:0/251549797 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6380c9940 0x7fa6381680a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.836+0000 7fa63fb41640 1 -- 172.21.15.117:0/251549797 >> 172.21.15.117:0/251549797 conn(0x7fa6380cc940 msgr2=0x7fa6380c8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.835+0000 7fa01764d640 1 -- start start 2024-09-17T05:27:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa01764d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa010172cc0 0x7fa0100c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa01764d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa01015ba90 0x7fa01015bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa01764d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 0x7fa010174b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa01764d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa0100c8a40 con 0x7fa010172cc0 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa01764d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa010074a00 con 0x7fa010174720 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa01764d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa0100c94a0 con 0x7fa01015ba90 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa015bea640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 0x7fa010174b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa015bea640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 0x7fa010174b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49000/0 (socket says 172.21.15.117:49000) 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.837+0000 7fa015bea640 1 -- 172.21.15.117:0/1337007272 learned_addr learned my addr 172.21.15.117:0/1337007272 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa0153e9640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa010172cc0 0x7fa0100c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa014be8640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa01015ba90 0x7fa01015bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa015bea640 1 -- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa01015ba90 msgr2=0x7fa01015bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa015bea640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa01015ba90 0x7fa01015bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa015bea640 1 -- 172.21.15.117:0/1337007272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa010172cc0 msgr2=0x7fa0100c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa015bea640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa010172cc0 0x7fa0100c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa015bea640 1 -- 172.21.15.117:0/1337007272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0101725e0 con 0x7fa010174720 2024-09-17T05:27:01.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.836+0000 7fa63fb41640 1 -- 172.21.15.117:0/251549797 shutdown_connections 2024-09-17T05:27:01.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.836+0000 7fa63fb41640 1 -- 172.21.15.117:0/251549797 wait complete. 2024-09-17T05:27:01.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.838+0000 7fa015bea640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 0x7fa010174b30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa0100c3320 tx=0x7fa000009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.840+0000 7fa63fb41640 1 Processor -- start 2024-09-17T05:27:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 -- 172.21.15.117:0/3660893194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d40c9940 msgr2=0x7f18d41680a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 --2- 172.21.15.117:0/3660893194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d40c9940 0x7f18d41680a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009720 tx=0x7f18c402b700 comp rx=0 tx=0).stop 2024-09-17T05:27:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 -- 172.21.15.117:0/3660893194 shutdown_connections 2024-09-17T05:27:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 --2- 172.21.15.117:0/3660893194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d40c9940 0x7f18d41680a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 -- 172.21.15.117:0/3660893194 >> 172.21.15.117:0/3660893194 conn(0x7f18d40cc940 msgr2=0x7f18d40c8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f3a2324d640 1 -- 172.21.15.117:0/159353207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f39fc005580 con 0x7f3a1c072e60 2024-09-17T05:27:01.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 -- 172.21.15.117:0/3660893194 shutdown_connections 2024-09-17T05:27:01.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.842+0000 7f18db08b640 1 -- 172.21.15.117:0/3660893194 wait complete. 2024-09-17T05:27:01.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.845+0000 7f18db08b640 1 Processor -- start 2024-09-17T05:27:01.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.847+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa000014820 con 0x7fa010174720 2024-09-17T05:27:01.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.847+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa00001a070 con 0x7fa010174720 2024-09-17T05:27:01.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.847+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa000014d70 con 0x7fa010174720 2024-09-17T05:27:01.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.845+0000 7f18db08b640 1 -- start start 2024-09-17T05:27:01.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.847+0000 7f18db08b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 0x7f18d4089aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.840+0000 7fa63fb41640 1 -- start start 2024-09-17T05:27:01.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.847+0000 7fa01764d640 1 -- 172.21.15.117:0/1337007272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0101728c0 con 0x7fa010174720 2024-09-17T05:27:01.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.847+0000 7fa01764d640 1 -- 172.21.15.117:0/1337007272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa010175b10 con 0x7fa010174720 2024-09-17T05:27:01.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.849+0000 7f3a197fa640 1 -- 172.21.15.117:0/159353207 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v10)=0 v10) ==== 127+0+0 (secure 0 0 0) 0x7f3a10039070 con 0x7f3a1c072e60 2024-09-17T05:27:01.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.849+0000 7fa63fb41640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6380739b0 0x7fa63815d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.849+0000 7fa63fb41640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa638167610 0x7fa638160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.849+0000 7fa63fb41640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 0x7fa6381611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 0x7fa6381611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 0x7fa6381611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38976/0 (socket says 172.21.15.117:38976) 2024-09-17T05:27:01.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7f01e612b640 1 -- 172.21.15.117:0/4047786215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e015bd20 msgr2=0x7f01e00c01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7f01e612b640 1 --2- 172.21.15.117:0/4047786215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e015bd20 0x7f01e00c01e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f01e00ca8e0 tx=0x7f01d802a540 comp rx=0 tx=0).stop 2024-09-17T05:27:01.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.851+0000 7f01e612b640 1 -- 172.21.15.117:0/4047786215 shutdown_connections 2024-09-17T05:27:01.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.851+0000 7f01e612b640 1 --2- 172.21.15.117:0/4047786215 >> v2:172.21.15.117:3300/0 conn(0x7f01e0074310 0x7f01e0074720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.851+0000 7f01e612b640 1 --2- 172.21.15.117:0/4047786215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e015bd20 0x7f01e00c01e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f01e00ca8e0 tx=0x7f01d802a540 comp rx=0 tx=0).stop 2024-09-17T05:27:01.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.851+0000 7f01e612b640 1 -- 172.21.15.117:0/4047786215 >> 172.21.15.117:0/4047786215 conn(0x7f01e00c9a20 msgr2=0x7f01e00c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 -- 172.21.15.117:0/1930737185 learned_addr learned my addr 172.21.15.117:0/1930737185 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380c8a40 con 0x7fa6381669b0 2024-09-17T05:27:01.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa638074a00 con 0x7fa6380739b0 2024-09-17T05:27:01.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa63806d4b0 con 0x7fa638167610 2024-09-17T05:27:01.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 -- 172.21.15.117:0/1930737185 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa638167610 msgr2=0x7fa638160c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa638167610 0x7fa638160c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 -- 172.21.15.117:0/1930737185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6380739b0 msgr2=0x7fa63815d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6380739b0 0x7fa63815d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.850+0000 7fa63e0de640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63808ba50 con 0x7fa6381669b0 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.851+0000 7fa63e0de640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 0x7fa6381611b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa62800b5b0 tx=0x7fa62800ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.852+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62800f660 con 0x7fa6381669b0 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.852+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa62800f910 con 0x7fa6381669b0 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.852+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6380c82d0 con 0x7fa6381669b0 2024-09-17T05:27:01.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.852+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa638165680 con 0x7fa6381669b0 2024-09-17T05:27:01.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.854+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62800fd60 con 0x7fa6381669b0 2024-09-17T05:27:01.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.854+0000 7f18d8e27640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 0x7f18d4089aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.854+0000 7f18d8e27640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 0x7f18d4089aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38980/0 (socket says 172.21.15.117:38980) 2024-09-17T05:27:01.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.854+0000 7f18d8e27640 1 -- 172.21.15.117:0/230120740 learned_addr learned my addr 172.21.15.117:0/230120740 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.851+0000 7f01e612b640 1 -- 172.21.15.117:0/4047786215 shutdown_connections 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.854+0000 7f18db08b640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18d4159e30 0x7f18d415adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18db08b640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d415c330 0x7f18d40c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d3fff640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18d4159e30 0x7f18d415adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d9628640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d415c330 0x7f18d40c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18db08b640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f18d40c8a40 con 0x7f18d40739b0 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18db08b640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f18d40747a0 con 0x7f18d4159e30 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18db08b640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f18d406d4b0 con 0x7f18d415c330 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d8e27640 1 -- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d415c330 msgr2=0x7f18d40c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d8e27640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d415c330 0x7f18d40c4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d8e27640 1 -- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18d4159e30 msgr2=0x7f18d415adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d8e27640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18d4159e30 0x7f18d415adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.855+0000 7f18d8e27640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d4089fe0 con 0x7f18d40739b0 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.856+0000 7f18d8e27640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 0x7f18d4089aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00cd20 tx=0x7f18cc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.856+0000 7f18d3fff640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18d4159e30 0x7f18d415adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.856+0000 7f18d9628640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d415c330 0x7f18d40c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.857+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc018070 con 0x7f18d40739b0 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.857+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18cc00ae50 con 0x7f18d40739b0 2024-09-17T05:27:01.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.857+0000 7f01e612b640 1 -- 172.21.15.117:0/4047786215 wait complete. 2024-09-17T05:27:01.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 -- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 msgr2=0x7f3a1c1674a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 0x7f3a1c1674a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3a10033100 tx=0x7f3a10003cd0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f18db08b640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d415b330 con 0x7f18d40739b0 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 -- 172.21.15.117:0/159353207 shutdown_connections 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a1c0c85d0 0x7f3a1c086060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a1c0c4d20 0x7f3a1c165e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 --2- 172.21.15.117:0/159353207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a1c072e60 0x7f3a1c1674a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 -- 172.21.15.117:0/159353207 >> 172.21.15.117:0/159353207 conn(0x7f3a1c0ccbe0 msgr2=0x7f3a1c0cb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 -- 172.21.15.117:0/159353207 shutdown_connections 2024-09-17T05:27:01.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f3a02ffd640 1 -- 172.21.15.117:0/159353207 wait complete. 2024-09-17T05:27:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.859+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa000003610 con 0x7fa010174720 2024-09-17T05:27:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.859+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa00001e890 con 0x7fa010174720 2024-09-17T05:27:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.859+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff0005290 con 0x7fa010174720 2024-09-17T05:27:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.858+0000 7f18db08b640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d415b900 con 0x7f18d40739b0 2024-09-17T05:27:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.860+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc006d70 con 0x7f18d40739b0 2024-09-17T05:27:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.860+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f18cc01b070 con 0x7f18d40739b0 2024-09-17T05:27:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.860+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f18cc007370 con 0x7f18d40739b0 2024-09-17T05:27:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.860+0000 7f01e612b640 1 Processor -- start 2024-09-17T05:27:01.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.860+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa628017400 con 0x7fa6381669b0 2024-09-17T05:27:01.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.860+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa62800dd70 con 0x7fa6381669b0 2024-09-17T05:27:01.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.861+0000 7fa623fff640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa618005290 con 0x7fa6381669b0 2024-09-17T05:27:01.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.862+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa000023020 con 0x7fa010174720 2024-09-17T05:27:01.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.861+0000 7f01e612b640 1 -- start start 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.863+0000 7f01e612b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 0x7f01e014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.863+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 0x7f01e014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.863+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 0x7f01e014d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38998/0 (socket says 172.21.15.117:38998) 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.863+0000 7f01df7fe640 1 -- 172.21.15.117:0/3924760894 learned_addr learned my addr 172.21.15.117:0/3924760894 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01e612b640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e00c8350 0x7f01e01fe950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01e612b640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f01e006dcc0 0x7f01e006e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f01e0073b10 con 0x7f01e0074310 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f01e0073830 con 0x7f01e00c8350 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f01e0074c60 con 0x7f01e006dcc0 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01df7fe640 1 -- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f01e006dcc0 msgr2=0x7f01e006e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01df7fe640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f01e006dcc0 0x7f01e006e0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01df7fe640 1 -- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e00c8350 msgr2=0x7f01e01fe950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01df7fe640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e00c8350 0x7f01e01fe950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.864+0000 7f01df7fe640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e01fee90 con 0x7f01e0074310 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.865+0000 7f01df7fe640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 0x7f01e014d510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.867+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8041030 con 0x7f01e0074310 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.867+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01d80315d0 con 0x7f01e0074310 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.867+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e015ee30 con 0x7f01e0074310 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.867+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e0152f70 con 0x7f01e0074310 2024-09-17T05:27:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.867+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8003c50 con 0x7f01e0074310 2024-09-17T05:27:01.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.868+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f01d8039040 con 0x7f01e0074310 2024-09-17T05:27:01.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.868+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f01d80467a0 con 0x7f01e0074310 2024-09-17T05:27:01.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.868+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa628013030 con 0x7fa6381669b0 2024-09-17T05:27:01.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.873+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c4005290 con 0x7f01e0074310 2024-09-17T05:27:01.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.873+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b4005290 con 0x7f18d40739b0 2024-09-17T05:27:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.879+0000 7fc3e5f52640 1 -- 172.21.15.117:0/1421304350 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 msgr2=0x7fc3d8035830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.879+0000 7fc3e5f52640 1 --2- 172.21.15.117:0/1421304350 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d8035830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc3e006c350 tx=0x7fc3e00a7470 comp rx=0 tx=0).stop 2024-09-17T05:27:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.879+0000 7fc3e5f52640 1 -- 172.21.15.117:0/1421304350 shutdown_connections 2024-09-17T05:27:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.879+0000 7fc3e5f52640 1 --2- 172.21.15.117:0/1421304350 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d8035830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.879+0000 7fc3e5f52640 1 --2- 172.21.15.117:0/1421304350 >> v2:172.21.15.117:3300/0 conn(0x7fc3d8007e30 0x7fc3d8045910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.879+0000 7fc3e5f52640 1 -- 172.21.15.117:0/1421304350 >> 172.21.15.117:0/1421304350 conn(0x7fc3d80506d0 msgr2=0x7fc3d8050ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.881+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f18cc005070 con 0x7f18d40739b0 2024-09-17T05:27:01.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.881+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f01d8003df0 con 0x7f01e0074310 2024-09-17T05:27:01.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.882+0000 7fc3e5f52640 1 -- 172.21.15.117:0/1421304350 shutdown_connections 2024-09-17T05:27:01.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.882+0000 7fc3e5f52640 1 -- 172.21.15.117:0/1421304350 wait complete. 2024-09-17T05:27:01.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.885+0000 7f71811da640 1 -- 172.21.15.117:0/3125740092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 msgr2=0x7f71740454b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.885+0000 7f71811da640 1 --2- 172.21.15.117:0/3125740092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740454b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7170005fd0 tx=0x7f717002a640 comp rx=0 tx=0).stop 2024-09-17T05:27:01.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.885+0000 7fc3e5f52640 1 Processor -- start 2024-09-17T05:27:01.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.885+0000 7fc3e5f52640 1 -- start start 2024-09-17T05:27:01.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7fc3e5f52640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 0x7fc3d80480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7f71811da640 1 -- 172.21.15.117:0/3125740092 shutdown_connections 2024-09-17T05:27:01.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7f71811da640 1 --2- 172.21.15.117:0/3125740092 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174005c00 0x7f7174006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7f71811da640 1 --2- 172.21.15.117:0/3125740092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740454b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7f71811da640 1 -- 172.21.15.117:0/3125740092 >> 172.21.15.117:0/3125740092 conn(0x7f717404fe30 msgr2=0x7f7174050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7fc3df7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 0x7fc3d80480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7fc3df7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 0x7fc3d80480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39042/0 (socket says 172.21.15.117:39042) 2024-09-17T05:27:01.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7fc3df7fe640 1 -- 172.21.15.117:0/63812519 learned_addr learned my addr 172.21.15.117:0/63812519 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.888+0000 7fc3e5f52640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d80cf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7f71811da640 1 -- 172.21.15.117:0/3125740092 shutdown_connections 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7fc3e5f52640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3d80496e0 0x7fc3d80dcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7fc3deffd640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d80cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7fc3dffff640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3d80496e0 0x7fc3d80dcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7fc3e5f52640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d8033d20 con 0x7fc3d8007e30 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7fc3e5f52640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d8033a40 con 0x7fc3d80496e0 2024-09-17T05:27:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7fc3e5f52640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d80dc770 con 0x7fc3d8035420 2024-09-17T05:27:01.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.889+0000 7f71811da640 1 -- 172.21.15.117:0/3125740092 wait complete. 2024-09-17T05:27:01.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.892+0000 7f71811da640 1 Processor -- start 2024-09-17T05:27:01.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.892+0000 7f71811da640 1 -- start start 2024-09-17T05:27:01.895 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi117') 2024-09-17T05:27:01.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.895+0000 7f71811da640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7174005c00 0x7f717417a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.895+0000 7f71811da640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174044d20 0x7f71740dd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.895+0000 7f717ad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7174005c00 0x7f717417a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.895+0000 7f717a59c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174044d20 0x7f71740dd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.896+0000 7f717a59c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174044d20 0x7f71740dd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:46602/0 (socket says 172.21.15.117:46602) 2024-09-17T05:27:01.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.896+0000 7f717ad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7174005c00 0x7f717417a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49078/0 (socket says 172.21.15.117:49078) 2024-09-17T05:27:01.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.896+0000 7f717a59c640 1 -- 172.21.15.117:0/3870586224 learned_addr learned my addr 172.21.15.117:0/3870586224 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.897+0000 7f71811da640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740d6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.897+0000 7f71811da640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f71740465d0 con 0x7f71740df620 2024-09-17T05:27:01.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.897+0000 7f71811da640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7174006550 con 0x7f7174005c00 2024-09-17T05:27:01.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 -- 172.21.15.117:0/2799021746 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 msgr2=0x7ff8080db6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 --2- 172.21.15.117:0/2799021746 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff8080db6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8080466b0 tx=0x7ff80402a530 comp rx=0 tx=0).stop 2024-09-17T05:27:01.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 -- 172.21.15.117:0/2799021746 shutdown_connections 2024-09-17T05:27:01.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 --2- 172.21.15.117:0/2799021746 >> v2:172.21.15.119:3300/0 conn(0x7ff8080d1690 0x7ff808005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 --2- 172.21.15.117:0/2799021746 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff8080db6b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 -- 172.21.15.117:0/2799021746 >> 172.21.15.117:0/2799021746 conn(0x7ff80804fe20 msgr2=0x7ff808050230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7f71811da640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f71740468b0 con 0x7f7174044d20 2024-09-17T05:27:01.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7fbc524bd640 1 -- 172.21.15.117:0/3626127153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc34003aa0 msgr2=0x7fbc34003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7fbc524bd640 1 --2- 172.21.15.117:0/3626127153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc34003aa0 0x7fbc34003f10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00a960 tx=0x7fbc34005830 comp rx=0 tx=0).stop 2024-09-17T05:27:01.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 -- 172.21.15.117:0/2799021746 shutdown_connections 2024-09-17T05:27:01.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.898+0000 7ff816c25640 1 -- 172.21.15.117:0/2799021746 wait complete. 2024-09-17T05:27:01.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fbc524bd640 1 -- 172.21.15.117:0/3626127153 shutdown_connections 2024-09-17T05:27:01.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fbc524bd640 1 --2- 172.21.15.117:0/3626127153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc34003aa0 0x7fbc34003f10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fbc524bd640 1 -- 172.21.15.117:0/3626127153 >> 172.21.15.117:0/3626127153 conn(0x7fbc4c0c9a20 msgr2=0x7fbc4c0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fbc524bd640 1 -- 172.21.15.117:0/3626127153 shutdown_connections 2024-09-17T05:27:01.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fbc524bd640 1 -- 172.21.15.117:0/3626127153 wait complete. 2024-09-17T05:27:01.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7fbc524bd640 1 Processor -- start 2024-09-17T05:27:01.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.901+0000 7ff816c25640 1 Processor -- start 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.901+0000 7ff816c25640 1 -- start start 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.904+0000 7ff816c25640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff808049880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.904+0000 7ff816c25640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8080469c0 0x7ff8080d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.904+0000 7ff816c25640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 0x7ff8080344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.904+0000 7ff816c25640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff808045ee0 con 0x7ff8080d1690 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.904+0000 7ff816c25640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff808045c00 con 0x7ff8080469c0 2024-09-17T05:27:01.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff816c25640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff8080461c0 con 0x7ff808044ca0 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 0x7ff8080344a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 0x7ff8080344a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39058/0 (socket says 172.21.15.117:39058) 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 -- 172.21.15.117:0/370208749 learned_addr learned my addr 172.21.15.117:0/370208749 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff80ffff640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8080469c0 0x7ff8080d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8149c1640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff808049880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7f717b59e640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7fbc524bd640 1 -- start start 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fc3df7fe640 1 -- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 msgr2=0x7fc3d80cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fc3df7fe640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d80cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fc3df7fe640 1 -- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3d80496e0 msgr2=0x7fc3d80dcbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fc3df7fe640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3d80496e0 0x7fc3d80dcbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.902+0000 7fc3df7fe640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d8048630 con 0x7fc3d8007e30 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7fc3df7fe640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 0x7fc3d80480f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400ea20 tx=0x7fc3d400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7fc3dffff640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3d80496e0 0x7fc3d80dcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7fc3deffd640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d80cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.907+0000 7fbc524bd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34003aa0 0x7fbc4c0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7f717b59e640 1 -- 172.21.15.117:0/3870586224 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174044d20 msgr2=0x7f71740dd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7f717b59e640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174044d20 0x7f71740dd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7f717b59e640 1 -- 172.21.15.117:0/3870586224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7174005c00 msgr2=0x7f717417a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7f717b59e640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7174005c00 0x7f717417a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.906+0000 7f717b59e640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71740dd680 con 0x7f71740df620 2024-09-17T05:27:01.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.907+0000 7f717b59e640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740d6080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f717c06b840 tx=0x7f717c06a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.907+0000 7fbc524bd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc4c15c800 0x7fbc4c201d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.907+0000 7fbc524bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 0x7fbc4c0be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34003aa0 0x7fbc4c0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34003aa0 0x7fbc4c0c00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49104/0 (socket says 172.21.15.117:49104) 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 -- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 msgr2=0x7ff808049880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff808049880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 -- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8080469c0 msgr2=0x7ff8080d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8080469c0 0x7ff8080d63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 -- 172.21.15.117:0/370208749 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff808049dc0 con 0x7ff8080d1690 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.905+0000 7ff8151c2640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 0x7ff8080344a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff810052860 tx=0x7ff81006a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7ff8149c1640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff808049880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81009a070 con 0x7ff8080d1690 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.912+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c074c30 con 0x7f71740df620 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.912+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f717c075400 con 0x7f71740df620 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.912+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c094650 con 0x7f71740df620 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc4b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc4c15c800 0x7fbc4c201d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc4b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc4c15c800 0x7fbc4c201d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:46638/0 (socket says 172.21.15.117:46638) 2024-09-17T05:27:01.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc50a5a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 0x7fbc4c0be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc50a5a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 0x7fbc4c0be9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39076/0 (socket says 172.21.15.117:39076) 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc524bd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c15c1e0 con 0x7fbc4c202250 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc524bd640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c15bf00 con 0x7fbc34003aa0 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc524bd640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c0c9270 con 0x7fbc4c15c800 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7fbc4bfff640 1 -- 172.21.15.117:0/395671794 learned_addr learned my addr 172.21.15.117:0/395671794 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.909+0000 7fbc50a5a640 1 -- 172.21.15.117:0/395671794 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc4c15c800 msgr2=0x7fbc4c201d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.909+0000 7fbc50a5a640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc4c15c800 0x7fbc4c201d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.909+0000 7fbc50a5a640 1 -- 172.21.15.117:0/395671794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34003aa0 msgr2=0x7fbc4c0c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.909+0000 7fbc50a5a640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34003aa0 0x7fbc4c0c00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.909+0000 7fbc50a5a640 1 -- 172.21.15.117:0/395671794 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c202a20 con 0x7fbc4c202250 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.909+0000 7fbc50a5a640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 0x7fbc4c0be9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c072d20 tx=0x7fbc4000d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.910+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40005020 con 0x7fbc4c202250 2024-09-17T05:27:01.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.910+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc400052b0 con 0x7fbc4c202250 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.910+0000 7fbc524bd640 1 -- 172.21.15.117:0/395671794 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c0c27b0 con 0x7fbc4c202250 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.910+0000 7fbc524bd640 1 -- 172.21.15.117:0/395671794 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c0c2e70 con 0x7fbc4c202250 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.912+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400047a0 con 0x7fbc4c202250 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff810074920 con 0x7ff8080d1690 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7ff816c25640 1 -- 172.21.15.117:0/370208749 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8080e7360 con 0x7ff8080d1690 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.908+0000 7ff816c25640 1 -- 172.21.15.117:0/370208749 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff808048640 con 0x7ff8080d1690 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.910+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810077230 con 0x7ff8080d1690 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff81009d070 con 0x7ff8080d1690 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff81009fdc0 con 0x7ff8080d1690 2024-09-17T05:27:01.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.912+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f0005290 con 0x7ff8080d1690 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 -- 172.21.15.117:0/2008703768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80463f0 msgr2=0x7f7fb80e3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 --2- 172.21.15.117:0/2008703768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80463f0 0x7f7fb80e3b50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d620 tx=0x7f7fc00a7510 comp rx=0 tx=0).stop 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 -- 172.21.15.117:0/2008703768 shutdown_connections 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 --2- 172.21.15.117:0/2008703768 >> v2:172.21.15.117:3300/0 conn(0x7f7fb80dd170 0x7f7fb804afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 --2- 172.21.15.117:0/2008703768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80463f0 0x7f7fb80e3b50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 --2- 172.21.15.117:0/2008703768 >> v2:172.21.15.119:3300/0 conn(0x7f7fb8045490 0x7f7fb8005c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.911+0000 7f7fc75c3640 1 -- 172.21.15.117:0/2008703768 >> 172.21.15.117:0/2008703768 conn(0x7f7fb804fe30 msgr2=0x7f7fb8050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.912+0000 7f7fc75c3640 1 -- 172.21.15.117:0/2008703768 shutdown_connections 2024-09-17T05:27:01.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.913+0000 7f71811da640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71740de4f0 con 0x7f71740df620 2024-09-17T05:27:01.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.914+0000 7f71811da640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71740deab0 con 0x7f71740df620 2024-09-17T05:27:01.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.915+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71740de260 con 0x7f71740df620 2024-09-17T05:27:01.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.916+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2c005290 con 0x7fbc4c202250 2024-09-17T05:27:01.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.916+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f717c09b060 con 0x7f71740df620 2024-09-17T05:27:01.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.916+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbc4001b020 con 0x7fbc4c202250 2024-09-17T05:27:01.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.916+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f717c09db50 con 0x7f71740df620 2024-09-17T05:27:01.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.917+0000 7f513ee15640 1 -- 172.21.15.117:0/2005473125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 msgr2=0x7f51300e2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.917+0000 7f513ee15640 1 --2- 172.21.15.117:0/2005473125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 0x7f51300e2820 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5130007160 tx=0x7f512402a4d0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.918+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbc4001d410 con 0x7fbc4c202250 2024-09-17T05:27:01.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.917+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f809c005580 con 0x7f80bc074cf0 2024-09-17T05:27:01.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.916+0000 7f7fc75c3640 1 -- 172.21.15.117:0/2008703768 wait complete. 2024-09-17T05:27:01.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.919+0000 7f7fc75c3640 1 Processor -- start 2024-09-17T05:27:01.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.920+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff810074cd0 con 0x7ff8080d1690 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.919+0000 7f7fc75c3640 1 -- start start 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.921+0000 7f7fc75c3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8045490 0x7f7fb8033de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.921+0000 7f7fc75c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 0x7f7fb80d0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.921+0000 7f7fc75c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80dd170 0x7f7fb804ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.921+0000 7f7fc75c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb80dd580 con 0x7f7fb80463f0 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.921+0000 7f7fc75c3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb80e3000 con 0x7f7fb8045490 2024-09-17T05:27:01.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.921+0000 7f7fc75c3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb80dd700 con 0x7f7fb80dd170 2024-09-17T05:27:01.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.920+0000 7f513ee15640 1 -- 172.21.15.117:0/2005473125 shutdown_connections 2024-09-17T05:27:01.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.920+0000 7f513ee15640 1 --2- 172.21.15.117:0/2005473125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 0x7f51300e2820 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.920+0000 7f513ee15640 1 --2- 172.21.15.117:0/2005473125 >> v2:172.21.15.119:3300/0 conn(0x7f5130047db0 0x7f51300452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.920+0000 7f513ee15640 1 -- 172.21.15.117:0/2005473125 >> 172.21.15.117:0/2005473125 conn(0x7f5130050010 msgr2=0x7f5130050420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.922+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4011330 con 0x7fc3d8007e30 2024-09-17T05:27:01.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.922+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc3d40159e0 con 0x7fc3d8007e30 2024-09-17T05:27:01.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.923+0000 7f7fc4b5e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 0x7f7fb80d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.923+0000 7f7fc4b5e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 0x7f7fb80d0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39110/0 (socket says 172.21.15.117:39110) 2024-09-17T05:27:01.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.923+0000 7f7fc4b5e640 1 -- 172.21.15.117:0/3226290816 learned_addr learned my addr 172.21.15.117:0/3226290816 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.922+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4015dd0 con 0x7fc3d8007e30 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.923+0000 7fc3e5f52640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d8175cf0 con 0x7fc3d8007e30 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7fc3e5f52640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d81762b0 con 0x7fc3d8007e30 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7f513ee15640 1 -- 172.21.15.117:0/2005473125 shutdown_connections 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7f7fc4b5e640 1 -- 172.21.15.117:0/3226290816 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80dd170 msgr2=0x7f7fb804ad40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7f7fc4b5e640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80dd170 0x7f7fb804ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7f7fc4b5e640 1 -- 172.21.15.117:0/3226290816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8045490 msgr2=0x7f7fb8033de0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7f7fc4b5e640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8045490 0x7f7fb8033de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.924+0000 7f7fc4b5e640 1 -- 172.21.15.117:0/3226290816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb80d8f50 con 0x7f7fb80463f0 2024-09-17T05:27:01.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.925+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d8000b90 con 0x7fc3d8007e30 2024-09-17T05:27:01.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.925+0000 7f513ee15640 1 -- 172.21.15.117:0/2005473125 wait complete. 2024-09-17T05:27:01.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.928+0000 7f7fc4b5e640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 0x7f7fb80d0410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0000b60 tx=0x7f7fc00aeee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.928+0000 7f513ee15640 1 Processor -- start 2024-09-17T05:27:01.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.928+0000 7f513ee15640 1 -- start start 2024-09-17T05:27:01.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513ee15640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 0x7f51300df2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513ee15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 0x7f51300d2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513ee15640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f51300dd390 0x7f51300cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513ee15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f51300457e0 con 0x7f51300e2410 2024-09-17T05:27:01.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513ee15640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51300481c0 con 0x7f5130047db0 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513ee15640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f513004c620 con 0x7f51300dd390 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513cbb1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 0x7f51300df2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513cbb1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 0x7f51300df2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49134/0 (socket says 172.21.15.117:49134) 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f513cbb1640 1 -- 172.21.15.117:0/2602371080 learned_addr learned my addr 172.21.15.117:0/2602371080 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.931+0000 7f5137fff640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 0x7f51300d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f513cbb1640 1 -- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f51300dd390 msgr2=0x7f51300cea30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f513cbb1640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f51300dd390 0x7f51300cea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f513cbb1640 1 -- 172.21.15.117:0/2602371080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 msgr2=0x7f51300d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f513cbb1640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 0x7f51300d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f513cbb1640 1 -- 172.21.15.117:0/2602371080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51300dd0c0 con 0x7f5130047db0 2024-09-17T05:27:01.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f513cbb1640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 0x7f51300df2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b420 tx=0x7f512c00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.932+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c005920 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.933+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51300df7e0 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.933+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51300dcde0 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.934+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f512c005bd0 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.934+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c006020 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.935+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f512c018020 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.936+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f512c01a550 con 0x7f5130047db0 2024-09-17T05:27:01.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.936+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5118005290 con 0x7f5130047db0 2024-09-17T05:27:01.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.936+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fbc40010670 con 0x7fbc4c202250 2024-09-17T05:27:01.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.943+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc00ad970 con 0x7f7fb80463f0 2024-09-17T05:27:01.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.943+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f717c0a4080 con 0x7f71740df620 2024-09-17T05:27:01.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.943+0000 7f7fc75c3640 1 -- 172.21.15.117:0/3226290816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb816c8a0 con 0x7f7fb80463f0 2024-09-17T05:27:01.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.943+0000 7f7fc75c3640 1 -- 172.21.15.117:0/3226290816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb816b200 con 0x7f7fb80463f0 2024-09-17T05:27:01.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.948+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f512c006760 con 0x7f5130047db0 2024-09-17T05:27:01.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.958+0000 7f7fc75c3640 1 -- 172.21.15.117:0/3226290816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb8005c00 con 0x7f7fb80463f0 2024-09-17T05:27:01.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.959+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc3d400c970 con 0x7fc3d8007e30 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.960+0000 7f21e632b640 1 -- 172.21.15.117:0/793632499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8005c30 msgr2=0x7f21d8006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.960+0000 7f21e632b640 1 --2- 172.21.15.117:0/793632499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8005c30 0x7f21d8006040 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f21d4009910 tx=0x7f21d403b720 comp rx=0 tx=0).stop 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.960+0000 7f21e632b640 1 -- 172.21.15.117:0/793632499 shutdown_connections 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.960+0000 7f21e632b640 1 --2- 172.21.15.117:0/793632499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8005c30 0x7f21d8006040 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.960+0000 7f21e632b640 1 --2- 172.21.15.117:0/793632499 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d8044d50 0x7f21d80454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.961+0000 7f21e632b640 1 -- 172.21.15.117:0/793632499 >> 172.21.15.117:0/793632499 conn(0x7f21d804fe60 msgr2=0x7f21d8050270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.959+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc3d401e400 con 0x7fc3d8007e30 2024-09-17T05:27:01.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.959+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fc00ae7c0 con 0x7f7fb80463f0 2024-09-17T05:27:01.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.959+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc00add30 con 0x7f7fb80463f0 2024-09-17T05:27:01.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.961+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7fc00b8410 con 0x7f7fb80463f0 2024-09-17T05:27:01.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.963+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc3d8000b90 con 0x7fc3d8007e30 2024-09-17T05:27:01.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.963+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7fc00c0740 con 0x7f7fb80463f0 2024-09-17T05:27:01.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.966+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7fc00cf080 con 0x7f7fb80463f0 2024-09-17T05:27:01.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.972+0000 7f80a3fff640 1 -- 172.21.15.117:0/3391369988 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v11)=0 v11) ==== 116+0+0 (secure 0 0 0) 0x7f80ac009470 con 0x7f80bc074cf0 2024-09-17T05:27:01.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.963+0000 7f21e632b640 1 -- 172.21.15.117:0/793632499 shutdown_connections 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.963+0000 7f21e632b640 1 -- 172.21.15.117:0/793632499 wait complete. 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.966+0000 7f21e632b640 1 Processor -- start 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.966+0000 7f21e632b640 1 -- start start 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.968+0000 7f21e632b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21d8005c30 0x7f21d8192fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.968+0000 7f21e632b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8044d50 0x7f21d80d9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.968+0000 7f21e632b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d80df650 0x7f21d80dba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.968+0000 7f21e632b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8046600 con 0x7f21d8044d50 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.968+0000 7f21e632b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8006580 con 0x7f21d8005c30 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.968+0000 7f21e632b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f21d80468e0 con 0x7f21d80df650 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.971+0000 7f21dffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21d8005c30 0x7f21d8192fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.971+0000 7f21dffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21d8005c30 0x7f21d8192fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49144/0 (socket says 172.21.15.117:49144) 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.971+0000 7f21dffff640 1 -- 172.21.15.117:0/2815128781 learned_addr learned my addr 172.21.15.117:0/2815128781 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.972+0000 7f21e48c8640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d80df650 0x7f21d80dba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f21e48c8640 1 -- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21d8005c30 msgr2=0x7f21d8192fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f21e48c8640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21d8005c30 0x7f21d8192fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f21e48c8640 1 -- 172.21.15.117:0/2815128781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8044d50 msgr2=0x7f21d80d9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f21e48c8640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8044d50 0x7f21d80d9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f21e48c8640 1 -- 172.21.15.117:0/2815128781 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d81965d0 con 0x7f21d80df650 2024-09-17T05:27:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f21e48c8640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d80df650 0x7f21d80dba10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f21d4004000 tx=0x7f21d4004d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 msgr2=0x7f80bc1529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 0x7f80bc1529a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00ed60 tx=0x7f80ac00c6a0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 shutdown_connections 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc15b220 0x7f80bc148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80bc1551b0 0x7f80bc150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 --2- 172.21.15.117:0/3391369988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80bc074cf0 0x7f80bc1529a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.973+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 >> 172.21.15.117:0/3391369988 conn(0x7f80bc0ccbe0 msgr2=0x7f80bc0c4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.974+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 shutdown_connections 2024-09-17T05:27:01.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.974+0000 7f80a1ffb640 1 -- 172.21.15.117:0/3391369988 wait complete. 2024-09-17T05:27:01.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.978+0000 7f48faf58640 1 -- 172.21.15.117:0/3484324191 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f48d4005850 con 0x7f48ec044cb0 2024-09-17T05:27:01.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.980+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d4014070 con 0x7f21d80df650 2024-09-17T05:27:01.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.980+0000 7f21e632b640 1 -- 172.21.15.117:0/2815128781 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d8197350 con 0x7f21d80df650 2024-09-17T05:27:01.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.980+0000 7f21e632b640 1 -- 172.21.15.117:0/2815128781 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d8196b50 con 0x7f21d80df650 2024-09-17T05:27:01.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.984+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21d4003e40 con 0x7f21d80df650 2024-09-17T05:27:01.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.984+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d4005490 con 0x7f21d80df650 2024-09-17T05:27:01.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.988+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f21d4017070 con 0x7f21d80df650 2024-09-17T05:27:01.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.988+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f21d4010960 con 0x7f21d80df650 2024-09-17T05:27:01.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.993+0000 7f48f17fa640 1 -- 172.21.15.117:0/3484324191 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v12)=0 v12) ==== 116+0+0 (secure 0 0 0) 0x7f48e0004b70 con 0x7f48ec044cb0 2024-09-17T05:27:01.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.993+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c0005290 con 0x7f21d80df650 2024-09-17T05:27:01.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:01.998+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f21d4004420 con 0x7f21d80df650 2024-09-17T05:27:02.002 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 -- 172.21.15.117:0/3484324191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 msgr2=0x7f48ec049890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 0x7f48ec049890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48e000ca30 tx=0x7f48e000cef0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 -- 172.21.15.117:0/3484324191 shutdown_connections 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48ec0469d0 0x7f48ec0c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec044cb0 0x7f48ec049890 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 --2- 172.21.15.117:0/3484324191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec00de50 0x7f48ec161d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.003+0000 7f48deffd640 1 -- 172.21.15.117:0/3484324191 >> 172.21.15.117:0/3484324191 conn(0x7f48ec04fe30 msgr2=0x7f48ec0d9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.004+0000 7f48deffd640 1 -- 172.21.15.117:0/3484324191 shutdown_connections 2024-09-17T05:27:02.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.004+0000 7f48deffd640 1 -- 172.21.15.117:0/3484324191 wait complete. 2024-09-17T05:27:02.047 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.070+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fe9a0005850 con 0x7fe9b8044d50 2024-09-17T05:27:02.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.070+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7fed0815e450 con 0x7fed0815b220 2024-09-17T05:27:02.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.076+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f63140336e0 con 0x7f631404d260 2024-09-17T05:27:02.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.076+0000 7fe9a77fe640 1 -- 172.21.15.117:0/3705834645 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v13)=0 v13) ==== 119+0+0 (secure 0 0 0) 0x7fe9b0003bd0 con 0x7fe9b8044d50 2024-09-17T05:27:02.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.084+0000 7f6311ffb640 1 -- 172.21.15.117:0/3180130072 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v14)=0 v14) ==== 115+0+0 (secure 0 0 0) 0x7f630c006d70 con 0x7f631404d260 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f631404d260 msgr2=0x7f63140de8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f631404d260 0x7f63140de8c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f630c00c3c0 tx=0x7f630c00c880 comp rx=0 tx=0).stop 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 shutdown_connections 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63140df490 0x7f63140c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63140d2590 0x7f63140c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 --2- 172.21.15.117:0/3180130072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f631404d260 0x7f63140de8c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 >> 172.21.15.117:0/3180130072 conn(0x7f6314050f50 msgr2=0x7f631404f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.087+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 shutdown_connections 2024-09-17T05:27:02.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.087+0000 7f63077fe640 1 -- 172.21.15.117:0/3180130072 wait complete. 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 msgr2=0x7fe9b804e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 0x7fe9b804e260 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000c520 tx=0x7fe9b000c9e0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 shutdown_connections 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9b80df650 0x7fe9b80cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b8044d50 0x7fe9b804e260 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 --2- 172.21.15.117:0/3705834645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9b8005c30 0x7fe9b80d5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.086+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 >> 172.21.15.117:0/3705834645 conn(0x7fe9b804fe60 msgr2=0x7fe9b80d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.087+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 shutdown_connections 2024-09-17T05:27:02.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.087+0000 7fe9a57fa640 1 -- 172.21.15.117:0/3705834645 wait complete. 2024-09-17T05:27:02.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.108+0000 7fecff7fe640 1 -- 172.21.15.117:0/2934591313 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v15)=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7fed081fea60 con 0x7fed0815b220 2024-09-17T05:27:02.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 msgr2=0x7fed08153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 0x7fed08153f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fed0000b9c0 tx=0x7fed0000be80 comp rx=0 tx=0).stop 2024-09-17T05:27:02.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 shutdown_connections 2024-09-17T05:27:02.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed0815b220 0x7fed08153f30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed08074cd0 0x7fed080c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 --2- 172.21.15.117:0/2934591313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed0806cf80 0x7fed0815dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.119+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 >> 172.21.15.117:0/2934591313 conn(0x7fed080ccbe0 msgr2=0x7fed08159050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.120+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 shutdown_connections 2024-09-17T05:27:02.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.120+0000 7fecfd7fa640 1 -- 172.21.15.117:0/2934591313 wait complete. 2024-09-17T05:27:02.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.136+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f76b8005580 con 0x7f76d815e4b0 2024-09-17T05:27:02.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.140+0000 7f76bffff640 1 -- 172.21.15.117:0/2995019052 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v16)=0 v16) ==== 121+0+0 (secure 0 0 0) 0x7f76c8031860 con 0x7f76d815e4b0 2024-09-17T05:27:02.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.143+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f18b4005850 con 0x7f18d40739b0 2024-09-17T05:27:02.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.148+0000 7f18d17fa640 1 -- 172.21.15.117:0/230120740 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v17)=0 v17) ==== 120+0+0 (secure 0 0 0) 0x7f18cc005250 con 0x7f18d40739b0 2024-09-17T05:27:02.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 msgr2=0x7f76d80c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 0x7f76d80c2950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f76c800baf0 tx=0x7f76c802f670 comp rx=0 tx=0).stop 2024-09-17T05:27:02.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 shutdown_connections 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f76d814f390 0x7f76d814f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76d81f8b10 0x7f76d81f8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 --2- 172.21.15.117:0/2995019052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d815e4b0 0x7f76d80c2950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 >> 172.21.15.117:0/2995019052 conn(0x7f76d80ccb40 msgr2=0x7f76d8159710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 shutdown_connections 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.149+0000 7f76bdffb640 1 -- 172.21.15.117:0/2995019052 wait complete. 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.150+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 msgr2=0x7f18d4089aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.150+0000 7f18baffd640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 0x7f18d4089aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00cd20 tx=0x7f18cc00a430 comp rx=0 tx=0).stop 2024-09-17T05:27:02.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.155+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f5254005850 con 0x7f52640de6b0 2024-09-17T05:27:02.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.156+0000 7f01e612b640 1 -- 172.21.15.117:0/3924760894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f01c4005580 con 0x7f01e0074310 2024-09-17T05:27:02.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.156+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 shutdown_connections 2024-09-17T05:27:02.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.156+0000 7f18baffd640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f18d415c330 0x7f18d40c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.156+0000 7f18baffd640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18d4159e30 0x7f18d415adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.156+0000 7f18baffd640 1 --2- 172.21.15.117:0/230120740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18d40739b0 0x7f18d4089aa0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.156+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 >> 172.21.15.117:0/230120740 conn(0x7f18d40cc940 msgr2=0x7f18d4163560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.157+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 shutdown_connections 2024-09-17T05:27:02.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.157+0000 7f18baffd640 1 -- 172.21.15.117:0/230120740 wait complete. 2024-09-17T05:27:02.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.160+0000 7fa623fff640 1 -- 172.21.15.117:0/1930737185 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fa618005850 con 0x7fa6381669b0 2024-09-17T05:27:02.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.176+0000 7f01d7fff640 1 -- 172.21.15.117:0/3924760894 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v18)=0 v18) ==== 119+0+0 (secure 0 0 0) 0x7f01d8005540 con 0x7f01e0074310 2024-09-17T05:27:02.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 -- 172.21.15.117:0/3924760894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 msgr2=0x7f01e014d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 0x7f01e014d510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8003820 comp rx=0 tx=0).stop 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 -- 172.21.15.117:0/3924760894 shutdown_connections 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f01e006dcc0 0x7f01e006e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01e00c8350 0x7f01e01fe950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 --2- 172.21.15.117:0/3924760894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01e0074310 0x7f01e014d510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 -- 172.21.15.117:0/3924760894 >> 172.21.15.117:0/3924760894 conn(0x7f01e00c9a20 msgr2=0x7f01e01597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.179+0000 7f01d5ffb640 1 -- 172.21.15.117:0/3924760894 shutdown_connections 2024-09-17T05:27:02.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.180+0000 7f01d5ffb640 1 -- 172.21.15.117:0/3924760894 wait complete. 2024-09-17T05:27:02.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.183+0000 7f52627fc640 1 -- 172.21.15.117:0/90101550 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v19)=0 v19) ==== 118+0+0 (secure 0 0 0) 0x7f525c011e20 con 0x7f52640de6b0 2024-09-17T05:27:02.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.184+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52640de6b0 msgr2=0x7f52640c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.184+0000 7f52729eb640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52640de6b0 0x7f52640c9410 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f525c00b410 tx=0x7f525c00b8d0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.187+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 shutdown_connections 2024-09-17T05:27:02.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.187+0000 7f52729eb640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52640de6b0 0x7f52640c9410 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.187+0000 7f52729eb640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52640d5040 0x7f52640cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.187+0000 7f52729eb640 1 --2- 172.21.15.117:0/90101550 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f526404bbe0 0x7f52640da240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.187+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 >> 172.21.15.117:0/90101550 conn(0x7f5264050710 msgr2=0x7f52640cdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.188+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 shutdown_connections 2024-09-17T05:27:02.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.188+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7ff7f0005580 con 0x7ff8080d1690 2024-09-17T05:27:02.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.188+0000 7f52729eb640 1 -- 172.21.15.117:0/90101550 wait complete. 2024-09-17T05:27:02.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.192+0000 7fa62e7fc640 1 -- 172.21.15.117:0/1930737185 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v20)=0 v20) ==== 120+0+0 (secure 0 0 0) 0x7fa62800d400 con 0x7fa6381669b0 2024-09-17T05:27:02.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 msgr2=0x7fa6381611b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 0x7fa6381611b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa62800b5b0 tx=0x7fa62800ba70 comp rx=0 tx=0).stop 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 shutdown_connections 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6381669b0 0x7fa6381611b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa638167610 0x7fa638160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 --2- 172.21.15.117:0/1930737185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6380739b0 0x7fa63815d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 >> 172.21.15.117:0/1930737185 conn(0x7fa6380cc940 msgr2=0x7fa6381f0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 shutdown_connections 2024-09-17T05:27:02.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.193+0000 7fa63fb41640 1 -- 172.21.15.117:0/1930737185 wait complete. 2024-09-17T05:27:02.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.207+0000 7f7fc75c3640 1 -- 172.21.15.117:0/3226290816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f7fb80e3e40 con 0x7f7fb80463f0 2024-09-17T05:27:02.209 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.209 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.209 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.209 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.209 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_dir_prefetch', 'True'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.210+0000 7ff80d7fa640 1 -- 172.21.15.117:0/370208749 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v21)=0 v21) ==== 125+0+0 (secure 0 0 0) 0x7ff81009f440 con 0x7ff8080d1690 2024-09-17T05:27:02.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 msgr2=0x7ff8080344a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 0x7ff8080344a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff810052860 tx=0x7ff81006a0a0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 shutdown_connections 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8080d1690 0x7ff8080344a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8080469c0 0x7ff8080d63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 --2- 172.21.15.117:0/370208749 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff808044ca0 0x7ff808049880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.212+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 >> 172.21.15.117:0/370208749 conn(0x7ff80804fe20 msgr2=0x7ff80804c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.213+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 shutdown_connections 2024-09-17T05:27:02.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.213+0000 7ff7faffd640 1 -- 172.21.15.117:0/370208749 wait complete. 2024-09-17T05:27:02.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fc3d804ce40 con 0x7fc3d8007e30 2024-09-17T05:27:02.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.214+0000 7f7fb5ffb640 1 -- 172.21.15.117:0/3226290816 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v22)=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7f7fc00bc030 con 0x7f7fb80463f0 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3226290816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 msgr2=0x7f7fb80d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7f7fab7fe640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 0x7f7fb80d0410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0000b60 tx=0x7f7fc00aeee0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3226290816 shutdown_connections 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7f7fab7fe640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7fb80dd170 0x7f7fb804ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7f7fab7fe640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fb80463f0 0x7f7fb80d0410 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0000b60 tx=0x7f7fc00aeee0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.216+0000 7f7fab7fe640 1 --2- 172.21.15.117:0/3226290816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8045490 0x7f7fb8033de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.217+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3226290816 >> 172.21.15.117:0/3226290816 conn(0x7f7fb804fe30 msgr2=0x7f7fb80063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.217+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f5118005580 con 0x7f5130047db0 2024-09-17T05:27:02.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.217+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3226290816 shutdown_connections 2024-09-17T05:27:02.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.217+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3226290816 wait complete. 2024-09-17T05:27:02.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.217+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f71740dca10 con 0x7f71740df620 2024-09-17T05:27:02.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.225+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fbc2c005580 con 0x7fbc4c202250 2024-09-17T05:27:02.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.230+0000 7fc3cbfff640 1 -- 172.21.15.117:0/63812519 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v23)=0 v23) ==== 106+0+0 (secure 0 0 0) 0x7fc3d402d080 con 0x7fc3d8007e30 2024-09-17T05:27:02.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 msgr2=0x7fc3d80480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 0x7fc3d80480f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400ea20 tx=0x7fc3d400eee0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 shutdown_connections 2024-09-17T05:27:02.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3d80496e0 0x7fc3d80dcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc3d8035420 0x7fc3d80cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 --2- 172.21.15.117:0/63812519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d8007e30 0x7fc3d80480f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 >> 172.21.15.117:0/63812519 conn(0x7fc3d80506d0 msgr2=0x7fc3d80d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 shutdown_connections 2024-09-17T05:27:02.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.232+0000 7fc3c9ffb640 1 -- 172.21.15.117:0/63812519 wait complete. 2024-09-17T05:27:02.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.235+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mds_dir_prefetch}] v 0) -- 0x7f9ff0005580 con 0x7fa010174720 2024-09-17T05:27:02.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.237+0000 7f51357fa640 1 -- 172.21.15.117:0/2602371080 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v24)=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f512c01a7a0 con 0x7f5130047db0 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 msgr2=0x7f51300df2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 0x7f51300df2a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b420 tx=0x7f512c00b8e0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 shutdown_connections 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f51300dd390 0x7f51300cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300e2410 0x7f51300d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 --2- 172.21.15.117:0/2602371080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5130047db0 0x7f51300df2a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 >> 172.21.15.117:0/2602371080 conn(0x7f5130050010 msgr2=0x7f5130048690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 shutdown_connections 2024-09-17T05:27:02.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.238+0000 7f513ee15640 1 -- 172.21.15.117:0/2602371080 wait complete. 2024-09-17T05:27:02.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.239+0000 7f71637fe640 1 -- 172.21.15.117:0/3870586224 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v25)=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7f717c09dd50 con 0x7f71740df620 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 msgr2=0x7f71740d6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740d6080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f717c06b840 tx=0x7f717c06a0a0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 shutdown_connections 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71740df620 0x7f71740d6080 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7174044d20 0x7f71740dd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 --2- 172.21.15.117:0/3870586224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7174005c00 0x7f717417a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 >> 172.21.15.117:0/3870586224 conn(0x7f717404fe30 msgr2=0x7f717404dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 shutdown_connections 2024-09-17T05:27:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.241+0000 7f71617fa640 1 -- 172.21.15.117:0/3870586224 wait complete. 2024-09-17T05:27:02.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.244+0000 7fbc48ff9640 1 -- 172.21.15.117:0/395671794 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v26)=0 v26) ==== 111+0+0 (secure 0 0 0) 0x7fbc40010850 con 0x7fbc4c202250 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 msgr2=0x7fbc4c0be9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 0x7fbc4c0be9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c072d20 tx=0x7fbc4000d9a0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 shutdown_connections 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc4c202250 0x7fbc4c0be9b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc4c15c800 0x7fbc4c201d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 --2- 172.21.15.117:0/395671794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34003aa0 0x7fbc4c0c00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 >> 172.21.15.117:0/395671794 conn(0x7fbc4c0c9a20 msgr2=0x7fbc4c1516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 shutdown_connections 2024-09-17T05:27:02.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.246+0000 7fbc327fc640 1 -- 172.21.15.117:0/395671794 wait complete. 2024-09-17T05:27:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.250+0000 7fa005ffb640 1 -- 172.21.15.117:0/1337007272 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_dir_prefetch}]=0 v27)=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7fa000011070 con 0x7fa010174720 2024-09-17T05:27:02.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 msgr2=0x7fa010174b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 0x7fa010174b30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa0100c3320 tx=0x7fa000009a00 comp rx=0 tx=0).stop 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 shutdown_connections 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa010174720 0x7fa010174b30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa01015ba90 0x7fa01015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 --2- 172.21.15.117:0/1337007272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa010172cc0 0x7fa0100c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 >> 172.21.15.117:0/1337007272 conn(0x7fa0100cc940 msgr2=0x7fa010068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 shutdown_connections 2024-09-17T05:27:02.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.252+0000 7f9ffb7fe640 1 -- 172.21.15.117:0/1337007272 wait complete. 2024-09-17T05:27:02.266 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.266 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.267 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.267 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.267 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.267 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.266+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7f21c0005580 con 0x7f21d80df650 2024-09-17T05:27:02.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.273+0000 7f21dcff9640 1 -- 172.21.15.117:0/2815128781 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v28)=0 v28) ==== 108+0+0 (secure 0 0 0) 0x7f21d4005630 con 0x7f21d80df650 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.273+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d80df650 msgr2=0x7f21d80dba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.273+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d80df650 0x7f21d80dba10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f21d4004000 tx=0x7f21d4004d90 comp rx=0 tx=0).stop 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 shutdown_connections 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21d80df650 0x7f21d80dba10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21d8044d50 0x7f21d80d9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/2815128781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21d8005c30 0x7f21d8192fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 >> 172.21.15.117:0/2815128781 conn(0x7f21d804fe60 msgr2=0x7f21d804f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 shutdown_connections 2024-09-17T05:27:02.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.274+0000 7f21ca7fc640 1 -- 172.21.15.117:0/2815128781 wait complete. 2024-09-17T05:27:02.286 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.286 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.286 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.286 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi117') 2024-09-17T05:27:02.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config dump 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 -- 172.21.15.117:0/1352610865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 msgr2=0x7f5ed4152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 --2- 172.21.15.117:0/1352610865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4152210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40ce5b0 tx=0x7f5ec402a530 comp rx=0 tx=0).stop 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 -- 172.21.15.117:0/1352610865 shutdown_connections 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 --2- 172.21.15.117:0/1352610865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4152210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 -- 172.21.15.117:0/1352610865 >> 172.21.15.117:0/1352610865 conn(0x7f5ed406e590 msgr2=0x7f5ed40cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 -- 172.21.15.117:0/1352610865 shutdown_connections 2024-09-17T05:27:02.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.434+0000 7f5edc320640 1 -- 172.21.15.117:0/1352610865 wait complete. 2024-09-17T05:27:02.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.436+0000 7f5edc320640 1 Processor -- start 2024-09-17T05:27:02.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.436+0000 7f5edc320640 1 -- start start 2024-09-17T05:27:02.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5edc320640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ed4071850 0x7f5ed41f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5edc320640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ed40c5660 0x7f5ed4154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5edc320640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5edc320640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed40c7ff0 con 0x7f5ed40c67a0 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5edc320640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed40ce0d0 con 0x7f5ed4071850 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5edc320640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed40c83a0 con 0x7f5ed40c5660 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4147e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39146/0 (socket says 172.21.15.117:39146) 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 -- 172.21.15.117:0/4121441305 learned_addr learned my addr 172.21.15.117:0/4121441305 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda0bc640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ed4071850 0x7f5ed41f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5ed98bb640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ed40c5660 0x7f5ed4154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:02.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 -- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ed40c5660 msgr2=0x7f5ed4154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ed40c5660 0x7f5ed4154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 -- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ed4071850 msgr2=0x7f5ed41f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ed4071850 0x7f5ed41f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 -- 172.21.15.117:0/4121441305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed414df80 con 0x7f5ed40c67a0 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.437+0000 7f5eda8bd640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4147e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00ba10 tx=0x7f5ecc00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc019070 con 0x7f5ed40c67a0 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ecc00cc30 con 0x7f5ed40c67a0 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc00ce60 con 0x7f5ed40c67a0 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed414e260 con 0x7f5ed40c67a0 2024-09-17T05:27:02.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed414a970 con 0x7f5ed40c67a0 2024-09-17T05:27:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ecc01c070 con 0x7f5ed40c67a0 2024-09-17T05:27:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.438+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ecc004360 con 0x7f5ed40c67a0 2024-09-17T05:27:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.439+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed414e4a0 con 0x7f5ed40c67a0 2024-09-17T05:27:02.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.441+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5ed414e4a0 con 0x7f5ed40c67a0 2024-09-17T05:27:02.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.501+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f5ed40c7940 con 0x7f5ed40c67a0 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.501+0000 7f5ecaffd640 1 -- 172.21.15.117:0/4121441305 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v28) ==== 59+0+1848 (secure 0 0 0) 0x7f5ecc0046b0 con 0x7f5ed40c67a0 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:mgr advanced client_mount_timeout 30 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:mgr advanced debug_client 20/20 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:mgr advanced debug_mgr 20/20 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:mgr advanced debug_ms 1/1 2024-09-17T05:27:02.503 INFO:teuthology.orchestra.run.smithi117.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced debug_mds 20/20 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced debug_ms 2/2 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced mds_bal_fragment_size_max 10000 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced mds_bal_merge_size 5 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced mds_bal_split_bits 3 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced mds_bal_split_size 100 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds dev mds_debug_frag true 2024-09-17T05:27:02.504 INFO:teuthology.orchestra.run.smithi117.stdout:mds dev mds_debug_scatterstat true 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced mds_dir_prefetch true 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:mds dev mds_verify_scatter true 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:client advanced client_mount_timeout 600 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:client advanced debug_client 20/20 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:client advanced debug_ms 1/1 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T05:27:02.505 INFO:teuthology.orchestra.run.smithi117.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 msgr2=0x7f5ed4147e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4147e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00ba10 tx=0x7f5ecc00bed0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 shutdown_connections 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ed40c67a0 0x7f5ed4147e20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ed40c5660 0x7f5ed4154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 --2- 172.21.15.117:0/4121441305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ed4071850 0x7f5ed41f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 >> 172.21.15.117:0/4121441305 conn(0x7f5ed406e590 msgr2=0x7f5ed414fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 shutdown_connections 2024-09-17T05:27:02.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.502+0000 7f5edc320640 1 -- 172.21.15.117:0/4121441305 wait complete. 2024-09-17T05:27:02.514 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T05:27:02.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T05:27:02.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.624+0000 7f9020830640 1 -- 172.21.15.117:0/4180361711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c073dd0 msgr2=0x7f901c152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.624+0000 7f9020830640 1 --2- 172.21.15.117:0/4180361711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c073dd0 0x7f901c152f20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f900c009830 tx=0x7f900c02a510 comp rx=0 tx=0).stop 2024-09-17T05:27:02.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.624+0000 7f9020830640 1 -- 172.21.15.117:0/4180361711 shutdown_connections 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.624+0000 7f9020830640 1 --2- 172.21.15.117:0/4180361711 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f901c0c8a00 0x7f901c0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.624+0000 7f9020830640 1 --2- 172.21.15.117:0/4180361711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c073dd0 0x7f901c152f20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.624+0000 7f9020830640 1 -- 172.21.15.117:0/4180361711 >> 172.21.15.117:0/4180361711 conn(0x7f901c06e590 msgr2=0x7f901c0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.625+0000 7f9020830640 1 -- 172.21.15.117:0/4180361711 shutdown_connections 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.625+0000 7f9020830640 1 -- 172.21.15.117:0/4180361711 wait complete. 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.626+0000 7f9020830640 1 Processor -- start 2024-09-17T05:27:02.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.626+0000 7f9020830640 1 -- start start 2024-09-17T05:27:02.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f901c073dd0 0x7f901c153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:02.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c0c8a00 0x7f901c0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:02.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 0x7f901c14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:02.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f901c0c9dc0 con 0x7f901c159110 2024-09-17T05:27:02.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f901c071e40 con 0x7f901c0c8a00 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f901c0c9f40 con 0x7f901c073dd0 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 0x7f901c14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 0x7f901c14db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39162/0 (socket says 172.21.15.117:39162) 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 -- 172.21.15.117:0/2047541621 learned_addr learned my addr 172.21.15.117:0/2047541621 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901a59c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f901c073dd0 0x7f901c153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9019d9b640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c0c8a00 0x7f901c0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 -- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f901c073dd0 msgr2=0x7f901c153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f901c073dd0 0x7f901c153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 -- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c0c8a00 msgr2=0x7f901c0ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c0c8a00 0x7f901c0ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 -- 172.21.15.117:0/2047541621 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901c0cd0f0 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f901ad9d640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 0x7f901c14db00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f901400ec50 tx=0x7f901400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014003b70 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9014003d10 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f901c1ef440 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014004280 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.628+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f901c1efa00 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.629+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9014018470 con 0x7f901c159110 2024-09-17T05:27:02.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.629+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9014010880 con 0x7f901c159110 2024-09-17T05:27:02.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.629+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f901c1585e0 con 0x7f901c159110 2024-09-17T05:27:02.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.631+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f9014038860 con 0x7f901c159110 2024-09-17T05:27:02.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.694+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f901c071a20 con 0x7f901c159110 2024-09-17T05:27:02.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.696+0000 7f9012ffd640 1 -- 172.21.15.117:0/2047541621 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f9014026070 con 0x7f901c159110 2024-09-17T05:27:02.697 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-09-17T05:27:02.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 msgr2=0x7f901c14db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:02.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 0x7f901c14db00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f901400ec50 tx=0x7f901400c6a0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 shutdown_connections 2024-09-17T05:27:02.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901c159110 0x7f901c14db00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f901c0c8a00 0x7f901c0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 --2- 172.21.15.117:0/2047541621 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f901c073dd0 0x7f901c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:02.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 >> 172.21.15.117:0/2047541621 conn(0x7f901c06e590 msgr2=0x7f901c157990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:02.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 shutdown_connections 2024-09-17T05:27:02.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:02.697+0000 7f9020830640 1 -- 172.21.15.117:0/2047541621 wait complete. 2024-09-17T05:27:02.709 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T05:27:02.709 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T05:27:02.709 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:27:02.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T05:27:02.733 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:27:02.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T05:27:02.798 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:27:02.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T05:27:02.864 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:27:02.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T05:27:02.929 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:27:02.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T05:27:02.954 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:27:02.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T05:27:03.019 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:27:03.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-17T05:27:03.084 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-17T05:27:03.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-17T05:27:03.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 83a189b8-5217-4789-a097-5a1555923f5d 0 2024-09-17T05:27:03.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 -- 172.21.15.119:0/3611502375 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb40b56c0 msgr2=0x7f1bb4165c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 --2- 172.21.15.119:0/3611502375 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb40b56c0 0x7f1bb4165c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1bac005fd0 tx=0x7f1bac02a940 comp rx=0 tx=0).stop 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 -- 172.21.15.119:0/3611502375 shutdown_connections 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 --2- 172.21.15.119:0/3611502375 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb40b56c0 0x7f1bb4165c70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 --2- 172.21.15.119:0/3611502375 >> v2:172.21.15.117:3300/0 conn(0x7f1bb415bd70 0x7f1bb40ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 -- 172.21.15.119:0/3611502375 >> 172.21.15.119:0/3611502375 conn(0x7f1bb4074d80 msgr2=0x7f1bb40d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 -- 172.21.15.119:0/3611502375 shutdown_connections 2024-09-17T05:27:03.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.301+0000 7f1bb9307640 1 -- 172.21.15.119:0/3611502375 wait complete. 2024-09-17T05:27:03.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.303+0000 7f1bb9307640 1 Processor -- start 2024-09-17T05:27:03.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.303+0000 7f1bb9307640 1 -- start start 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.304+0000 7f1bb9307640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 0x7f1bb4153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.304+0000 7f1bb9307640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb40b56c0 0x7f1bb4152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb9307640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb415bd70 0x7f1bb4153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb9307640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb40cacb0 con 0x7f1bb40b56c0 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb9307640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb40cd0d0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb9307640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb41558b0 con 0x7f1bb415bd70 2024-09-17T05:27:03.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 0x7f1bb4153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 0x7f1bb4153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40878/0 (socket says 172.21.15.119:40878) 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 -- 172.21.15.119:0/2868872554 learned_addr learned my addr 172.21.15.119:0/2868872554 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb37fe640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb415bd70 0x7f1bb4153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb27fc640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb40b56c0 0x7f1bb4152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 -- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb415bd70 msgr2=0x7f1bb4153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb415bd70 0x7f1bb4153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 -- 172.21.15.119:0/2868872554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb40b56c0 msgr2=0x7f1bb4152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb40b56c0 0x7f1bb4152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 -- 172.21.15.119:0/2868872554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb40c6e00 con 0x7f1bb40b52b0 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb37fe640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb415bd70 0x7f1bb4153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb2ffd640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 0x7f1bb4153af0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400e790 tx=0x7f1ba400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:03.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400eea0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ba400cdb0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb40c70a0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4010690 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.305+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb41f1960 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.306+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1ba4010830 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.306+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f1ba4018df0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.306+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb40cc9f0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.308+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1bb40cc9f0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.379+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "83a189b8-5217-4789-a097-5a1555923f5d", "id": 0} v 0) -- 0x7f1bb40b61e0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.382+0000 7f1b9b7fe640 1 -- 172.21.15.119:0/2868872554 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83a189b8-5217-4789-a097-5a1555923f5d", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f1bb40b61e0 con 0x7f1bb40b52b0 2024-09-17T05:27:03.383 INFO:teuthology.orchestra.run.smithi119.stdout:0 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.383+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 msgr2=0x7f1bb4153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.383+0000 7f1bb9307640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 0x7f1bb4153af0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400e790 tx=0x7f1ba400ec50 comp rx=0 tx=0).stop 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 shutdown_connections 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1bb415bd70 0x7f1bb4153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb40b56c0 0x7f1bb4152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 --2- 172.21.15.119:0/2868872554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1bb40b52b0 0x7f1bb4153af0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 >> 172.21.15.119:0/2868872554 conn(0x7f1bb4074d80 msgr2=0x7f1bb40cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 shutdown_connections 2024-09-17T05:27:03.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.384+0000 7f1bb9307640 1 -- 172.21.15.119:0/2868872554 wait complete. 2024-09-17T05:27:03.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new fab753eb-635a-4a3a-b9aa-323ee34b0b92 1 2024-09-17T05:27:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 -- 172.21.15.119:0/3705013016 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf840cad30 msgr2=0x7fdf840c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 --2- 172.21.15.119:0/3705013016 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf840cad30 0x7fdf840c37c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009830 tx=0x7fdf7802a5c0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 -- 172.21.15.119:0/3705013016 shutdown_connections 2024-09-17T05:27:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 --2- 172.21.15.119:0/3705013016 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf840cad30 0x7fdf840c37c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 -- 172.21.15.119:0/3705013016 >> 172.21.15.119:0/3705013016 conn(0x7fdf84076430 msgr2=0x7fdf84072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 -- 172.21.15.119:0/3705013016 shutdown_connections 2024-09-17T05:27:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.507+0000 7fdf8aad5640 1 -- 172.21.15.119:0/3705013016 wait complete. 2024-09-17T05:27:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.508+0000 7fdf8aad5640 1 Processor -- start 2024-09-17T05:27:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.509+0000 7fdf8aad5640 1 -- start start 2024-09-17T05:27:03.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf8aad5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 0x7fdf8415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf8aad5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf84153260 0x7fdf8413a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf8aad5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8415d220 0x7fdf8414e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf8aad5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840cbc30 con 0x7fdf84153260 2024-09-17T05:27:03.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf8aad5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8415cd00 con 0x7fdf8415d220 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf8aad5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fdf840c9e40 con 0x7fdf840cad30 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf88871640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 0x7fdf8415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf89072640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8415d220 0x7fdf8414e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf88871640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 0x7fdf8415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48312/0 (socket says 172.21.15.119:48312) 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf88871640 1 -- 172.21.15.119:0/1099828195 learned_addr learned my addr 172.21.15.119:0/1099828195 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf83fff640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf84153260 0x7fdf8413a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf89072640 1 -- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 msgr2=0x7fdf8415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf89072640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 0x7fdf8415a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf89072640 1 -- 172.21.15.119:0/1099828195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf84153260 msgr2=0x7fdf8413a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf89072640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf84153260 0x7fdf8413a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf89072640 1 -- 172.21.15.119:0/1099828195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf841e3f60 con 0x7fdf8415d220 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.510+0000 7fdf88871640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 0x7fdf8415a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf89072640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8415d220 0x7fdf8414e400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00c9f0 tx=0x7fdf7c00ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c016020 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf7c00ac60 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8415a960 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c012680 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8415b220 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf7c019020 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.511+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fdf7c01bd50 con 0x7fdf8415d220 2024-09-17T05:27:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.512+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf84158400 con 0x7fdf8415d220 2024-09-17T05:27:03.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.516+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fdf84158400 con 0x7fdf8415d220 2024-09-17T05:27:03.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.584+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fab753eb-635a-4a3a-b9aa-323ee34b0b92", "id": 1} v 0) -- 0x7fdf8406fd40 con 0x7fdf8415d220 2024-09-17T05:27:03.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.589+0000 7fdf817fa640 1 -- 172.21.15.119:0/1099828195 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fab753eb-635a-4a3a-b9aa-323ee34b0b92", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fdf7c020030 con 0x7fdf8415d220 2024-09-17T05:27:03.589 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-17T05:27:03.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.589+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8415d220 msgr2=0x7fdf8414e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.589+0000 7fdf8aad5640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8415d220 0x7fdf8414e400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00c9f0 tx=0x7fdf7c00ceb0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 shutdown_connections 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8415d220 0x7fdf8414e400 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf84153260 0x7fdf8413a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 --2- 172.21.15.119:0/1099828195 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840cad30 0x7fdf8415a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 >> 172.21.15.119:0/1099828195 conn(0x7fdf84076430 msgr2=0x7fdf840c4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 shutdown_connections 2024-09-17T05:27:03.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.590+0000 7fdf8aad5640 1 -- 172.21.15.119:0/1099828195 wait complete. 2024-09-17T05:27:03.601 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 5c07e6d8-bcb2-481f-82ab-8500bfbf7719 2 2024-09-17T05:27:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/3101771324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 msgr2=0x7ff2c80741f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 --2- 172.21.15.119:0/3101771324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c80741f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0005fd0 tx=0x7ff2c002abc0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/3101771324 shutdown_connections 2024-09-17T05:27:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 --2- 172.21.15.119:0/3101771324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c80741f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/3101771324 >> 172.21.15.119:0/3101771324 conn(0x7ff2c80c9a20 msgr2=0x7ff2c80c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/3101771324 shutdown_connections 2024-09-17T05:27:03.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.711+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/3101771324 wait complete. 2024-09-17T05:27:03.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.713+0000 7ff2ccbc3640 1 Processor -- start 2024-09-17T05:27:03.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.713+0000 7ff2ccbc3640 1 -- start start 2024-09-17T05:27:03.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2ccbc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2c806deb0 0x7ff2c81f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2ccbc3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c8150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2ccbc3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff2c8152300 0x7ff2c806cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2ccbc3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c80c48d0 con 0x7ff2c806deb0 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2ccbc3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c815c1a0 con 0x7ff2c806e2c0 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2ccbc3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c80c4a50 con 0x7ff2c8152300 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2c5d9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c8150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2c6d9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff2c8152300 0x7ff2c806cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2c5d9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c8150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40912/0 (socket says 172.21.15.119:40912) 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2c5d9b640 1 -- 172.21.15.119:0/2437660567 learned_addr learned my addr 172.21.15.119:0/2437660567 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.714+0000 7ff2c659c640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2c806deb0 0x7ff2c81f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c6d9d640 1 -- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 msgr2=0x7ff2c8150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c6d9d640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c8150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c6d9d640 1 -- 172.21.15.119:0/2437660567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2c806deb0 msgr2=0x7ff2c81f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c6d9d640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2c806deb0 0x7ff2c81f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c6d9d640 1 -- 172.21.15.119:0/2437660567 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c81f5410 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c5d9b640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c8150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2c6d9d640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff2c8152300 0x7ff2c806cda0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0032000 tx=0x7ff2c00042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0043020 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2c00046f0 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c8153ba0 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c003e6b0 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c8153e40 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff2c0046020 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.715+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7ff2c0034840 con 0x7ff2c8152300 2024-09-17T05:27:03.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.716+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c80748a0 con 0x7ff2c8152300 2024-09-17T05:27:03.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.718+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff2c0034420 con 0x7ff2c8152300 2024-09-17T05:27:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.789+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5c07e6d8-bcb2-481f-82ab-8500bfbf7719", "id": 2} v 0) -- 0x7ff2c80c8ff0 con 0x7ff2c8152300 2024-09-17T05:27:03.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.793+0000 7ff2beffd640 1 -- 172.21.15.119:0/2437660567 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c07e6d8-bcb2-481f-82ab-8500bfbf7719", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7ff2c0034b60 con 0x7ff2c8152300 2024-09-17T05:27:03.794 INFO:teuthology.orchestra.run.smithi119.stdout:2 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff2c8152300 msgr2=0x7ff2c806cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff2c8152300 0x7ff2c806cda0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0032000 tx=0x7ff2c00042f0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 shutdown_connections 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff2c8152300 0x7ff2c806cda0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2c806e2c0 0x7ff2c8150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 --2- 172.21.15.119:0/2437660567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2c806deb0 0x7ff2c81f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 >> 172.21.15.119:0/2437660567 conn(0x7ff2c80c9a20 msgr2=0x7ff2c8156ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 shutdown_connections 2024-09-17T05:27:03.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.794+0000 7ff2ccbc3640 1 -- 172.21.15.119:0/2437660567 wait complete. 2024-09-17T05:27:03.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 50957aa1-aed2-438e-8f96-6c81bd139df7 3 2024-09-17T05:27:03.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 -- 172.21.15.119:0/836157684 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 msgr2=0x7fc420152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 --2- 172.21.15.119:0/836157684 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420152210 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc410005fd0 tx=0x7fc41002a540 comp rx=0 tx=0).stop 2024-09-17T05:27:03.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 -- 172.21.15.119:0/836157684 shutdown_connections 2024-09-17T05:27:03.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 --2- 172.21.15.119:0/836157684 >> v2:172.21.15.117:3300/0 conn(0x7fc420071850 0x7fc420071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 --2- 172.21.15.119:0/836157684 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420152210 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 -- 172.21.15.119:0/836157684 >> 172.21.15.119:0/836157684 conn(0x7fc42006e590 msgr2=0x7fc4200cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 -- 172.21.15.119:0/836157684 shutdown_connections 2024-09-17T05:27:03.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.916+0000 7fc42537b640 1 -- 172.21.15.119:0/836157684 wait complete. 2024-09-17T05:27:03.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.918+0000 7fc42537b640 1 Processor -- start 2024-09-17T05:27:03.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.918+0000 7fc42537b640 1 -- start start 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc42537b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc420071850 0x7fc4200cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc42537b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 0x7fc42015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc42537b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc42537b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4200c7ff0 con 0x7fc420071850 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc42537b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc4200ce0d0 con 0x7fc4200c4f50 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc42537b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc4200c83a0 con 0x7fc4200c67a0 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc41e7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 0x7fc42015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc41f7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc41e7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 0x7fc42015aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40922/0 (socket says 172.21.15.119:40922) 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc41f7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48334/0 (socket says 172.21.15.119:48334) 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc41effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc420071850 0x7fc4200cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.919+0000 7fc41e7fc640 1 -- 172.21.15.119:0/4108166795 learned_addr learned my addr 172.21.15.119:0/4108166795 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41f7fe640 1 -- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 msgr2=0x7fc42015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41f7fe640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 0x7fc42015aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41f7fe640 1 -- 172.21.15.119:0/4108166795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc420071850 msgr2=0x7fc4200cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41f7fe640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc420071850 0x7fc4200cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41f7fe640 1 -- 172.21.15.119:0/4108166795 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4200ccc50 con 0x7fc4200c67a0 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41e7fc640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 0x7fc42015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41effd640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc420071850 0x7fc4200cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc41f7fe640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420155460 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc418001e40 tx=0x7fc418010420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41800e0d0 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc41800e270 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4201ef710 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418010d90 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc42014f0e0 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.920+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc418003d00 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.921+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fc41800cb10 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.922+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4200c7940 con 0x7fc4200c67a0 2024-09-17T05:27:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.924+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc418026020 con 0x7fc4200c67a0 2024-09-17T05:27:03.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.992+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "50957aa1-aed2-438e-8f96-6c81bd139df7", "id": 3} v 0) -- 0x7fc4200c8d00 con 0x7fc4200c67a0 2024-09-17T05:27:03.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.996+0000 7fc4077fe640 1 -- 172.21.15.119:0/4108166795 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "50957aa1-aed2-438e-8f96-6c81bd139df7", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fc41800c680 con 0x7fc4200c67a0 2024-09-17T05:27:03.997 INFO:teuthology.orchestra.run.smithi119.stdout:3 2024-09-17T05:27:03.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 msgr2=0x7fc420155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:03.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420155460 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc418001e40 tx=0x7fc418010420 comp rx=0 tx=0).stop 2024-09-17T05:27:03.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 shutdown_connections 2024-09-17T05:27:03.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200c67a0 0x7fc420155460 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200c4f50 0x7fc42015aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 --2- 172.21.15.119:0/4108166795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc420071850 0x7fc4200cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 >> 172.21.15.119:0/4108166795 conn(0x7fc42006e590 msgr2=0x7fc42014e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 shutdown_connections 2024-09-17T05:27:03.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:03.997+0000 7fc42537b640 1 -- 172.21.15.119:0/4108166795 wait complete. 2024-09-17T05:27:04.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new acec14cf-20ec-49de-afeb-8b1fa3e3e602 4 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 -- 172.21.15.119:0/1858262591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f692406deb0 msgr2=0x7f69240c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 --2- 172.21.15.119:0/1858262591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f692406deb0 0x7f69240c6b10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6924158f20 tx=0x7f691402a670 comp rx=0 tx=0).stop 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 -- 172.21.15.119:0/1858262591 shutdown_connections 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 --2- 172.21.15.119:0/1858262591 >> v2:172.21.15.117:3300/0 conn(0x7f6924074010 0x7f6924074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 --2- 172.21.15.119:0/1858262591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f692406deb0 0x7f69240c6b10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 -- 172.21.15.119:0/1858262591 >> 172.21.15.119:0/1858262591 conn(0x7f69240c9a20 msgr2=0x7f69240c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 -- 172.21.15.119:0/1858262591 shutdown_connections 2024-09-17T05:27:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.120+0000 7f692a04a640 1 -- 172.21.15.119:0/1858262591 wait complete. 2024-09-17T05:27:04.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.122+0000 7f692a04a640 1 Processor -- start 2024-09-17T05:27:04.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.122+0000 7f692a04a640 1 -- start start 2024-09-17T05:27:04.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f692a04a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 0x7f6924157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f692a04a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6924074010 0x7f6924152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f692a04a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6924152100 0x7f69241532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f692a04a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6924074960 con 0x7f6924152100 2024-09-17T05:27:04.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f692a04a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6924158a30 con 0x7f6924074010 2024-09-17T05:27:04.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f692a04a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6924157ea0 con 0x7f692406deb0 2024-09-17T05:27:04.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f69237fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 0x7f6924157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f6923fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6924152100 0x7f69241532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f69237fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 0x7f6924157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48350/0 (socket says 172.21.15.119:48350) 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f69237fe640 1 -- 172.21.15.119:0/3234743875 learned_addr learned my addr 172.21.15.119:0/3234743875 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.123+0000 7f6922ffd640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6924074010 0x7f6924152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f69237fe640 1 -- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6924074010 msgr2=0x7f6924152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f69237fe640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6924074010 0x7f6924152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f69237fe640 1 -- 172.21.15.119:0/3234743875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6924152100 msgr2=0x7f69241532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f69237fe640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6924152100 0x7f69241532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f69237fe640 1 -- 172.21.15.119:0/3234743875 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69240cb720 con 0x7f692406deb0 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f69237fe640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 0x7f6924157640 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6914034040 tx=0x7f6914003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:04.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691403e030 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6914003d10 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914035040 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6924155340 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69240cc0d0 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6914041030 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.124+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f6914043d30 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.126+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6924076400 con 0x7f692406deb0 2024-09-17T05:27:04.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.128+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6914043410 con 0x7f692406deb0 2024-09-17T05:27:04.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.224+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "acec14cf-20ec-49de-afeb-8b1fa3e3e602", "id": 4} v 0) -- 0x7f6924070ce0 con 0x7f692406deb0 2024-09-17T05:27:04.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.228+0000 7f691bfff640 1 -- 172.21.15.119:0/3234743875 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "acec14cf-20ec-49de-afeb-8b1fa3e3e602", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f6914004f80 con 0x7f692406deb0 2024-09-17T05:27:04.228 INFO:teuthology.orchestra.run.smithi119.stdout:4 2024-09-17T05:27:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 msgr2=0x7f6924157640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 0x7f6924157640 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6914034040 tx=0x7f6914003820 comp rx=0 tx=0).stop 2024-09-17T05:27:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 shutdown_connections 2024-09-17T05:27:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6924152100 0x7f69241532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6924074010 0x7f6924152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 --2- 172.21.15.119:0/3234743875 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f692406deb0 0x7f6924157640 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 >> 172.21.15.119:0/3234743875 conn(0x7f69240c9a20 msgr2=0x7f69240c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 shutdown_connections 2024-09-17T05:27:04.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.229+0000 7f692a04a640 1 -- 172.21.15.119:0/3234743875 wait complete. 2024-09-17T05:27:04.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 37bb0708-41d4-4ce1-98bb-403acf42b745 5 2024-09-17T05:27:04.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 -- 172.21.15.119:0/2599895809 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 msgr2=0x7f7a7815abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2599895809 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a7815abf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7a78070920 tx=0x7f7a6802a560 comp rx=0 tx=0).stop 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 -- 172.21.15.119:0/2599895809 shutdown_connections 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2599895809 >> v2:172.21.15.119:3300/0 conn(0x7f7a7815b130 0x7f7a780b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2599895809 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a7815abf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2599895809 >> v2:172.21.15.117:3300/0 conn(0x7f7a780c6f50 0x7f7a78073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 -- 172.21.15.119:0/2599895809 >> 172.21.15.119:0/2599895809 conn(0x7f7a780c9a20 msgr2=0x7f7a780c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 -- 172.21.15.119:0/2599895809 shutdown_connections 2024-09-17T05:27:04.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.352+0000 7f7a7eade640 1 -- 172.21.15.119:0/2599895809 wait complete. 2024-09-17T05:27:04.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.354+0000 7f7a7eade640 1 Processor -- start 2024-09-17T05:27:04.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.354+0000 7f7a7eade640 1 -- start start 2024-09-17T05:27:04.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7eade640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a780c6f50 0x7f7a781efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7eade640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a780bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7eade640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a7815b130 0x7f7a780c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7eade640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78157dd0 con 0x7f7a780c6f50 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7eade640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78157af0 con 0x7f7a7815b130 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7eade640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f7a780c87a0 con 0x7f7a780c8390 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a780bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a780bfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48360/0 (socket says 172.21.15.119:48360) 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a6ffff640 1 -- 172.21.15.119:0/2456129268 learned_addr learned my addr 172.21.15.119:0/2456129268 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7d07b640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a7815b130 0x7f7a780c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7c87a640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a780c6f50 0x7f7a781efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7d07b640 1 -- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 msgr2=0x7f7a780bfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7d07b640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a780bfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7d07b640 1 -- 172.21.15.119:0/2456129268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a780c6f50 msgr2=0x7f7a781efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7d07b640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a780c6f50 0x7f7a781efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.355+0000 7f7a7d07b640 1 -- 172.21.15.119:0/2456129268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a781f0140 con 0x7f7a7815b130 2024-09-17T05:27:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a7d07b640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a7815b130 0x7f7a780c2ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a70005fd0 tx=0x7f7a7000c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7000cde0 con 0x7f7a7815b130 2024-09-17T05:27:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a70004c10 con 0x7f7a7815b130 2024-09-17T05:27:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70005020 con 0x7f7a7815b130 2024-09-17T05:27:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781509e0 con 0x7f7a7815b130 2024-09-17T05:27:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7a7001d020 con 0x7f7a7815b130 2024-09-17T05:27:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a781f0350 con 0x7f7a7815b130 2024-09-17T05:27:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.356+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f7a70016d50 con 0x7f7a7815b130 2024-09-17T05:27:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.357+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a780cb400 con 0x7f7a7815b130 2024-09-17T05:27:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.359+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7a780cb400 con 0x7f7a7815b130 2024-09-17T05:27:04.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.429+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "37bb0708-41d4-4ce1-98bb-403acf42b745", "id": 5} v 0) -- 0x7f7a78150730 con 0x7f7a7815b130 2024-09-17T05:27:04.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.433+0000 7f7a6d7fa640 1 -- 172.21.15.119:0/2456129268 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37bb0708-41d4-4ce1-98bb-403acf42b745", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f7a70024070 con 0x7f7a7815b130 2024-09-17T05:27:04.434 INFO:teuthology.orchestra.run.smithi119.stdout:5 2024-09-17T05:27:04.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a7815b130 msgr2=0x7f7a780c2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a7815b130 0x7f7a780c2ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a70005fd0 tx=0x7f7a7000c3d0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 shutdown_connections 2024-09-17T05:27:04.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a7815b130 0x7f7a780c2ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7a780c8390 0x7f7a780bfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 --2- 172.21.15.119:0/2456129268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a780c6f50 0x7f7a781efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 >> 172.21.15.119:0/2456129268 conn(0x7f7a780c9a20 msgr2=0x7f7a780cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 shutdown_connections 2024-09-17T05:27:04.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.434+0000 7f7a7eade640 1 -- 172.21.15.119:0/2456129268 wait complete. 2024-09-17T05:27:04.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 6c2aae50-c4c8-4f33-85ff-ebf6ebf40877 6 2024-09-17T05:27:04.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 -- 172.21.15.119:0/2661527372 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c40c8a00 msgr2=0x7f20c40c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 --2- 172.21.15.119:0/2661527372 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c40c8a00 0x7f20c40c6c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009be0 tx=0x7f20bc02a920 comp rx=0 tx=0).stop 2024-09-17T05:27:04.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 -- 172.21.15.119:0/2661527372 shutdown_connections 2024-09-17T05:27:04.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 --2- 172.21.15.119:0/2661527372 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c40c8a00 0x7f20c40c6c80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 -- 172.21.15.119:0/2661527372 >> 172.21.15.119:0/2661527372 conn(0x7f20c406e590 msgr2=0x7f20c40cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 -- 172.21.15.119:0/2661527372 shutdown_connections 2024-09-17T05:27:04.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.558+0000 7f20caf69640 1 -- 172.21.15.119:0/2661527372 wait complete. 2024-09-17T05:27:04.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.560+0000 7f20caf69640 1 Processor -- start 2024-09-17T05:27:04.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.560+0000 7f20caf69640 1 -- start start 2024-09-17T05:27:04.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20caf69640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c4073dd0 0x7f20c414f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20caf69640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20c40c8a00 0x7f20c4148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20caf69640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 0x7f20c41476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20caf69640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f20c4071e40 con 0x7f20c40c8a00 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20caf69640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f20c4153630 con 0x7f20c4073dd0 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20caf69640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f20c40c9dc0 con 0x7f20c4159110 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 0x7f20c41476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c8d05640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c4073dd0 0x7f20c414f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 0x7f20c41476f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48378/0 (socket says 172.21.15.119:48378) 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 -- 172.21.15.119:0/4117430798 learned_addr learned my addr 172.21.15.119:0/4117430798 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c3fff640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20c40c8a00 0x7f20c4148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 -- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c4073dd0 msgr2=0x7f20c414f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c4073dd0 0x7f20c414f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 -- 172.21.15.119:0/4117430798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20c40c8a00 msgr2=0x7f20c4148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20c40c8a00 0x7f20c4148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.561+0000 7f20c9506640 1 -- 172.21.15.119:0/4117430798 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c414fd30 con 0x7f20c4159110 2024-09-17T05:27:04.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20c9506640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 0x7f20c41476f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f20bc005fd0 tx=0x7f20bc003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:04.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc030e60 con 0x7f20c4159110 2024-09-17T05:27:04.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20bc03e070 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c4151d10 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc032330 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c4146780 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f20bc004040 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.562+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f20bc043a60 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.563+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c4146570 con 0x7f20c4159110 2024-09-17T05:27:04.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.565+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f20bc02d070 con 0x7f20c4159110 2024-09-17T05:27:04.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.650+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6c2aae50-c4c8-4f33-85ff-ebf6ebf40877", "id": 6} v 0) -- 0x7f20c4071b00 con 0x7f20c4159110 2024-09-17T05:27:04.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.653+0000 7f20c17fa640 1 -- 172.21.15.119:0/4117430798 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c2aae50-c4c8-4f33-85ff-ebf6ebf40877", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f20bc043d00 con 0x7f20c4159110 2024-09-17T05:27:04.654 INFO:teuthology.orchestra.run.smithi119.stdout:6 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 msgr2=0x7f20c41476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 0x7f20c41476f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f20bc005fd0 tx=0x7f20bc003c20 comp rx=0 tx=0).stop 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 shutdown_connections 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c4159110 0x7f20c41476f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20c40c8a00 0x7f20c4148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 --2- 172.21.15.119:0/4117430798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c4073dd0 0x7f20c414f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 >> 172.21.15.119:0/4117430798 conn(0x7f20c406e590 msgr2=0x7f20c414e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 shutdown_connections 2024-09-17T05:27:04.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.654+0000 7f20caf69640 1 -- 172.21.15.119:0/4117430798 wait complete. 2024-09-17T05:27:04.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 693ac895-4b40-4737-ba27-8ce2ef7743b1 7 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 -- 172.21.15.119:0/732041701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c4f50 msgr2=0x7f42400c5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 --2- 172.21.15.119:0/732041701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c4f50 0x7f42400c5360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f424015b660 tx=0x7f423802a780 comp rx=0 tx=0).stop 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 -- 172.21.15.119:0/732041701 shutdown_connections 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 --2- 172.21.15.119:0/732041701 >> v2:172.21.15.117:3300/0 conn(0x7f4240071850 0x7f4240071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 --2- 172.21.15.119:0/732041701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c4f50 0x7f42400c5360 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 --2- 172.21.15.119:0/732041701 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f42400c67a0 0x7f4240152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 -- 172.21.15.119:0/732041701 >> 172.21.15.119:0/732041701 conn(0x7f424006e590 msgr2=0x7f42400cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 -- 172.21.15.119:0/732041701 shutdown_connections 2024-09-17T05:27:04.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.778+0000 7f4245bda640 1 -- 172.21.15.119:0/732041701 wait complete. 2024-09-17T05:27:04.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.779+0000 7f4245bda640 1 Processor -- start 2024-09-17T05:27:04.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.779+0000 7f4245bda640 1 -- start start 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 0x7f424014ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42400c4f50 0x7f4240154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c67a0 0x7f424014d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42400ce0d0 con 0x7f42400c4f50 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f42400c58a0 con 0x7f42400c67a0 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f42400c7ff0 con 0x7f4240071850 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 0x7f424014ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 0x7f424014ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48392/0 (socket says 172.21.15.119:48392) 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 -- 172.21.15.119:0/3363557614 learned_addr learned my addr 172.21.15.119:0/3363557614 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-17T05:27:04.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423ffff640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c67a0 0x7f424014d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4237fff640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42400c4f50 0x7f4240154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 -- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c67a0 msgr2=0x7f424014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c67a0 0x7f424014d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 -- 172.21.15.119:0/3363557614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42400c4f50 msgr2=0x7f4240154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42400c4f50 0x7f4240154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 -- 172.21.15.119:0/3363557614 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42401554b0 con 0x7f4240071850 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423f7fe640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 0x7f424014ac70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f423000e9c0 tx=0x7f423000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423000cd80 con 0x7f4240071850 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4230004020 con 0x7f4240071850 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4240149ae0 con 0x7f4240071850 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230004450 con 0x7f4240071850 2024-09-17T05:27:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.781+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4240148fe0 con 0x7f4240071850 2024-09-17T05:27:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.782+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4230003980 con 0x7f4240071850 2024-09-17T05:27:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.782+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f4230018a90 con 0x7f4240071850 2024-09-17T05:27:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.782+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4240152540 con 0x7f4240071850 2024-09-17T05:27:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.784+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f4240152540 con 0x7f4240071850 2024-09-17T05:27:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.854+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "693ac895-4b40-4737-ba27-8ce2ef7743b1", "id": 7} v 0) -- 0x7f42400c8d00 con 0x7f4240071850 2024-09-17T05:27:04.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.859+0000 7f423cff9640 1 -- 172.21.15.119:0/3363557614 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "693ac895-4b40-4737-ba27-8ce2ef7743b1", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f4230009460 con 0x7f4240071850 2024-09-17T05:27:04.859 INFO:teuthology.orchestra.run.smithi119.stdout:7 2024-09-17T05:27:04.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 msgr2=0x7f424014ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 0x7f424014ac70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f423000e9c0 tx=0x7f423000ee80 comp rx=0 tx=0).stop 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 shutdown_connections 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42400c67a0 0x7f424014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42400c4f50 0x7f4240154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 --2- 172.21.15.119:0/3363557614 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4240071850 0x7f424014ac70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 >> 172.21.15.119:0/3363557614 conn(0x7f424006e590 msgr2=0x7f4240150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 shutdown_connections 2024-09-17T05:27:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-17T05:27:04.860+0000 7f4245bda640 1 -- 172.21.15.119:0/3363557614 wait complete. 2024-09-17T05:27:04.871 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T05:27:04.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T05:27:04.873 INFO:tasks.ceph.osd.0:Started 2024-09-17T05:27:04.873 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T05:27:04.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T05:27:04.875 INFO:tasks.ceph.osd.1:Started 2024-09-17T05:27:04.875 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T05:27:04.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T05:27:04.878 INFO:tasks.ceph.osd.2:Started 2024-09-17T05:27:04.878 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T05:27:04.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T05:27:04.880 INFO:tasks.ceph.osd.3:Started 2024-09-17T05:27:04.880 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T05:27:04.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T05:27:04.882 INFO:tasks.ceph.osd.4:Started 2024-09-17T05:27:04.882 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T05:27:04.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T05:27:04.885 INFO:tasks.ceph.osd.5:Started 2024-09-17T05:27:04.885 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-17T05:27:04.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-17T05:27:04.887 INFO:tasks.ceph.osd.6:Started 2024-09-17T05:27:04.887 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-17T05:27:04.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-17T05:27:04.889 INFO:tasks.ceph.osd.7:Started 2024-09-17T05:27:04.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:05.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.026+0000 7f117660e640 1 -- 172.21.15.117:0/840512672 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c5660 msgr2=0x7f1170163a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:05.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.026+0000 7f117660e640 1 --2- 172.21.15.117:0/840512672 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c5660 0x7f1170163a50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1168009be0 tx=0x7f116802a890 comp rx=0 tx=0).stop 2024-09-17T05:27:05.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.026+0000 7f117660e640 1 -- 172.21.15.117:0/840512672 shutdown_connections 2024-09-17T05:27:05.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.026+0000 7f117660e640 1 --2- 172.21.15.117:0/840512672 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c5660 0x7f1170163a50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.026+0000 7f117660e640 1 -- 172.21.15.117:0/840512672 >> 172.21.15.117:0/840512672 conn(0x7f117006e590 msgr2=0x7f11700cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:05.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.027+0000 7f117660e640 1 -- 172.21.15.117:0/840512672 shutdown_connections 2024-09-17T05:27:05.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.027+0000 7f117660e640 1 -- 172.21.15.117:0/840512672 wait complete. 2024-09-17T05:27:05.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.028+0000 7f117660e640 1 Processor -- start 2024-09-17T05:27:05.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.028+0000 7f117660e640 1 -- start start 2024-09-17T05:27:05.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 0x7f117015a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:05.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c67a0 0x7f11701efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:05.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1170152210 0x7f117014da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:05.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11700c7ff0 con 0x7f11700c5660 2024-09-17T05:27:05.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f11700ce0d0 con 0x7f1170152210 2024-09-17T05:27:05.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f11700c83a0 con 0x7f11700c67a0 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 0x7f117015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 0x7f117015a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58006/0 (socket says 172.21.15.117:58006) 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 -- 172.21.15.117:0/3917778207 learned_addr learned my addr 172.21.15.117:0/3917778207 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f1174bab640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1170152210 0x7f117014da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116f7fe640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c67a0 0x7f11701efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 -- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c67a0 msgr2=0x7f11701efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c67a0 0x7f11701efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 -- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1170152210 msgr2=0x7f117014da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1170152210 0x7f117014da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 -- 172.21.15.117:0/3917778207 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117014b7d0 con 0x7f11700c5660 2024-09-17T05:27:05.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116ffff640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 0x7f117015a680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f115800c410 tx=0x7f115800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158011790 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1158017070 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117014bab0 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.030+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158011cc0 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.031+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11701ef7b0 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.031+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f115801c460 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.031+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f115801b3c0 con 0x7f11700c5660 2024-09-17T05:27:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.032+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11701ef2c0 con 0x7f11700c5660 2024-09-17T05:27:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.034+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1158012740 con 0x7f11700c5660 2024-09-17T05:27:05.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.157+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11700c7940 con 0x7f11700c5660 2024-09-17T05:27:05.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.158+0000 7f116cff9640 1 -- 172.21.15.117:0/3917778207 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1158012920 con 0x7f11700c5660 2024-09-17T05:27:05.159 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:05.159 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:04.856691+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 msgr2=0x7f117015a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 0x7f117015a680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f115800c410 tx=0x7f115800c8d0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 shutdown_connections 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1170152210 0x7f117014da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11700c67a0 0x7f11701efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 --2- 172.21.15.117:0/3917778207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11700c5660 0x7f117015a680 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.159+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 >> 172.21.15.117:0/3917778207 conn(0x7f117006e590 msgr2=0x7f11701573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.160+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 shutdown_connections 2024-09-17T05:27:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:05.160+0000 7f117660e640 1 -- 172.21.15.117:0/3917778207 wait complete. 2024-09-17T05:27:05.175 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T05:27:05.175 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T05:27:05.985 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:06.091 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 -- 172.21.15.117:0/4198161576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240072350 msgr2=0x7fe2400b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 --2- 172.21.15.117:0/4198161576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240072350 0x7fe2400b1b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe230005fd0 tx=0x7fe23002a710 comp rx=0 tx=0).stop 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 -- 172.21.15.117:0/4198161576 shutdown_connections 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 --2- 172.21.15.117:0/4198161576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240072350 0x7fe2400b1b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 -- 172.21.15.117:0/4198161576 >> 172.21.15.117:0/4198161576 conn(0x7fe240069780 msgr2=0x7fe240072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 -- 172.21.15.117:0/4198161576 shutdown_connections 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.090+0000 7fe2479a8640 1 -- 172.21.15.117:0/4198161576 wait complete. 2024-09-17T05:27:06.092 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.091+0000 7fe2479a8640 1 Processor -- start 2024-09-17T05:27:06.093 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.091+0000 7fe2479a8640 1 -- start start 2024-09-17T05:27:06.095 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe240071be0 0x7fe2401f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:06.095 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe240072350 0x7fe2400c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 0x7fe240147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe2400716c0 con 0x7fe240152f20 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe2400713e0 con 0x7fe240071be0 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe2400c9170 con 0x7fe240072350 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 0x7fe240147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 0x7fe240147280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58012/0 (socket says 172.21.15.117:58012) 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 -- 172.21.15.117:0/2610150197 learned_addr learned my addr 172.21.15.117:0/2610150197 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245744640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe240071be0 0x7fe2401f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe244f43640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe240072350 0x7fe2400c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 -- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe240072350 msgr2=0x7fe2400c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe240072350 0x7fe2400c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 -- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe240071be0 msgr2=0x7fe2401f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe240071be0 0x7fe2401f1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 -- 172.21.15.117:0/2610150197 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2401f5b20 con 0x7fe240152f20 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe244f43640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe240072350 0x7fe2400c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe245f45640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 0x7fe240147280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe238002230 tx=0x7fe23800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238003b70 con 0x7fe240152f20 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe238003d10 con 0x7fe240152f20 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2401f4d60 con 0x7fe240152f20 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.093+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238005330 con 0x7fe240152f20 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.094+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2401f5320 con 0x7fe240152f20 2024-09-17T05:27:06.097 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.094+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 2) ==== 50742+0+0 (secure 0 0 0) 0x7fe2380056e0 con 0x7fe240152f20 2024-09-17T05:27:06.098 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.095+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe240153330 con 0x7fe240152f20 2024-09-17T05:27:06.098 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.095+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe2380521d0 con 0x7fe240152f20 2024-09-17T05:27:06.098 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.097+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe238012420 con 0x7fe240152f20 2024-09-17T05:27:06.120 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-17T05:27:06.118+0000 7fc1122eb640 -1 Falling back to public interface 2024-09-17T05:27:06.143 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-09-17T05:27:06.143+0000 7f4d0529c640 -1 Falling back to public interface 2024-09-17T05:27:06.147 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-09-17T05:27:06.146+0000 7f889222e640 -1 Falling back to public interface 2024-09-17T05:27:06.157 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.156+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2400c4750 con 0x7fe240152f20 2024-09-17T05:27:06.158 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.157+0000 7fe235ffb640 1 -- 172.21.15.117:0/2610150197 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe23800aa10 con 0x7fe240152f20 2024-09-17T05:27:06.158 INFO:teuthology.misc.health.smithi117.stdout: 2024-09-17T05:27:06.158 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:04.856691+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:06.159 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 msgr2=0x7fe240147280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:06.159 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 0x7fe240147280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe238002230 tx=0x7fe23800cb90 comp rx=0 tx=0).stop 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 shutdown_connections 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe240152f20 0x7fe240147280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe240072350 0x7fe2400c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 --2- 172.21.15.117:0/2610150197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe240071be0 0x7fe2401f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 >> 172.21.15.117:0/2610150197 conn(0x7fe240069780 msgr2=0x7fe2401480d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 shutdown_connections 2024-09-17T05:27:06.160 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:06.158+0000 7fe2479a8640 1 -- 172.21.15.117:0/2610150197 wait complete. 2024-09-17T05:27:06.168 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-17T05:27:06.354 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-09-17T05:27:06.353+0000 7efc3b725640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T05:27:06.354 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-09-17T05:27:06.353+0000 7efc3b725640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T05:27:06.356 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-09-17T05:27:06.355+0000 7f163be1b640 -1 Falling back to public interface 2024-09-17T05:27:06.375 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-17T05:27:06.374+0000 7f4aac0b6640 -1 Falling back to public interface 2024-09-17T05:27:06.377 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-09-17T05:27:06.376+0000 7f06b8830640 -1 Falling back to public interface 2024-09-17T05:27:06.403 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-09-17T05:27:06.402+0000 7fe00e416640 -1 Falling back to public interface 2024-09-17T05:27:06.588 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-09-17T05:27:06.588+0000 7fbd8d02a640 -1 Falling back to public interface 2024-09-17T05:27:09.676 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-09-17T05:27:09.675+0000 7f4d0529c640 -1 osd.5 0 log_to_monitors true 2024-09-17T05:27:10.128 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-09-17T05:27:10.127+0000 7fbd8d02a640 -1 osd.7 0 log_to_monitors true 2024-09-17T05:27:10.154 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-17T05:27:10.153+0000 7f4aac0b6640 -1 osd.6 0 log_to_monitors true 2024-09-17T05:27:10.182 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-09-17T05:27:10.181+0000 7f889222e640 -1 osd.4 0 log_to_monitors true 2024-09-17T05:27:10.203 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-17T05:27:10.201+0000 7fc1122eb640 -1 osd.2 0 log_to_monitors true 2024-09-17T05:27:10.356 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-09-17T05:27:10.355+0000 7f163be1b640 -1 osd.1 0 log_to_monitors true 2024-09-17T05:27:10.460 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-09-17T05:27:10.459+0000 7fe00e416640 -1 osd.3 0 log_to_monitors true 2024-09-17T05:27:10.909 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-09-17T05:27:10.907+0000 7f06b8830640 -1 osd.0 0 log_to_monitors true 2024-09-17T05:27:11.729 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-09-17T05:27:11.729+0000 7f4d0122a640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T05:27:11.748 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-09-17T05:27:11.747+0000 7f4cf9020640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:12.976 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:13.026 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-09-17T05:27:13.025+0000 7fbd897cd640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T05:27:13.036 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-17T05:27:13.035+0000 7f4aa8044640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T05:27:13.040 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-09-17T05:27:13.040+0000 7fbd80dae640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.051 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-09-17T05:27:13.051+0000 7f4a9fe3a640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.055 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-09-17T05:27:13.055+0000 7f888d839640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T05:27:13.066 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-09-17T05:27:13.066+0000 7f888562f640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.151 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.150+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/2021356925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 msgr2=0x7fa0f80749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:13.152 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.150+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/2021356925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80749a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80c6a40 tx=0x7fa0e802a650 comp rx=0 tx=0).stop 2024-09-17T05:27:13.157 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.153+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/2021356925 shutdown_connections 2024-09-17T05:27:13.157 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.153+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/2021356925 >> v2:172.21.15.119:3300/0 conn(0x7fa0f806f5f0 0x7fa0f806e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.157 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.153+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/2021356925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80749a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.157 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.153+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/2021356925 >> 172.21.15.117:0/2021356925 conn(0x7fa0f8070fa0 msgr2=0x7fa0f80c8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:13.161 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.157+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/2021356925 shutdown_connections 2024-09-17T05:27:13.161 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.157+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/2021356925 wait complete. 2024-09-17T05:27:13.161 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.159+0000 7fa0fdcf9640 1 Processor -- start 2024-09-17T05:27:13.165 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.160+0000 7fa0fdcf9640 1 -- start start 2024-09-17T05:27:13.165 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0fdcf9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0f806f5f0 0x7fa0f81fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:13.165 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0fdcf9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:13.165 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0fdcf9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0f815e800 0x7fa0f81473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:13.165 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0fdcf9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f8074ee0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0fdcf9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f8074480 con 0x7fa0f815e800 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0fdcf9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f8072e60 con 0x7fa0f806f5f0 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0f6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0f6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58054/0 (socket says 172.21.15.117:58054) 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.162+0000 7fa0f6ffd640 1 -- 172.21.15.117:0/63851976 learned_addr learned my addr 172.21.15.117:0/63851976 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.163+0000 7fa0f6ffd640 1 -- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0f806f5f0 msgr2=0x7fa0f81fecc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.163+0000 7fa0f6ffd640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0f806f5f0 0x7fa0f81fecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.163+0000 7fa0f6ffd640 1 -- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0f815e800 msgr2=0x7fa0f81473a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.163+0000 7fa0f6ffd640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0f815e800 0x7fa0f81473a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.163+0000 7fa0f6ffd640 1 -- 172.21.15.117:0/63851976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f81480f0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.163+0000 7fa0f6ffd640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80ca4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000bd60 tx=0x7fa0f0009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.164+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0019070 con 0x7fa0f80c6e00 2024-09-17T05:27:13.166 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.164+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f81ff200 con 0x7fa0f80c6e00 2024-09-17T05:27:13.167 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.164+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f80cb880 con 0x7fa0f80c6e00 2024-09-17T05:27:13.170 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.168+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0f000ccf0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.170 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.168+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00146a0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.170 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.168+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) ==== 101385+0+0 (secure 0 0 0) 0x7fa0f001c070 con 0x7fa0f80c6e00 2024-09-17T05:27:13.175 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.174+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f815b210 con 0x7fa0f80c6e00 2024-09-17T05:27:13.175 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.174+0000 7fa0dffff640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa0e4078c00 0x7fa0e407b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:13.176 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.175+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3351+0+0 (secure 0 0 0) 0x7fa0f009a2b0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.178 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.177+0000 7fa0f77fe640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa0e4078c00 0x7fa0e407b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:13.179 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.178+0000 7fa0f77fe640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa0e4078c00 0x7fa0e407b110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8033000 tx=0x7fa0e8004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:13.186 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.185+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa0f00630e0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.211 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-09-17T05:27:13.210+0000 7fe009de0640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T05:27:13.213 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-17T05:27:13.212+0000 7fc10ea8e640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T05:27:13.214 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-09-17T05:27:13.213+0000 7f1637807640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T05:27:13.217 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-09-17T05:27:13.216+0000 7f06b4586640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T05:27:13.226 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-09-17T05:27:13.225+0000 7fe001bd6640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.229 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-09-17T05:27:13.228+0000 7f162f5fd640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.232 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-17T05:27:13.231+0000 7fc10606f640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.232 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-09-17T05:27:13.231+0000 7f06abb67640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T05:27:13.383 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.382+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0f815c080 con 0x7fa0f80c6e00 2024-09-17T05:27:13.384 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.383+0000 7fa0dffff640 1 -- 172.21.15.117:0/63851976 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9048 (secure 0 0 0) 0x7fa0f00629e0 con 0x7fa0f80c6e00 2024-09-17T05:27:13.384 INFO:teuthology.misc.health.smithi117.stdout: 2024-09-17T05:27:13.384 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":14,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:13.107472+0000","last_up_change":"2024-09-17T05:27:13.107472+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:13.386 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa0e4078c00 msgr2=0x7fa0e407b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa0e4078c00 0x7fa0e407b110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8033000 tx=0x7fa0e8004730 comp rx=0 tx=0).stop 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 msgr2=0x7fa0f80ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80ca4a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000bd60 tx=0x7fa0f0009eb0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 shutdown_connections 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa0e4078c00 0x7fa0e407b110 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0f815e800 0x7fa0f81473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0f80c6e00 0x7fa0f80ca4a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 --2- 172.21.15.117:0/63851976 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0f806f5f0 0x7fa0f81fecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 >> 172.21.15.117:0/63851976 conn(0x7fa0f8070fa0 msgr2=0x7fa0f814f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 shutdown_connections 2024-09-17T05:27:13.387 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:13.385+0000 7fa0fdcf9640 1 -- 172.21.15.117:0/63851976 wait complete. 2024-09-17T05:27:13.396 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-09-17T05:27:20.202 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff473bad640 1 -- 172.21.15.117:0/3126884473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff458003f90 msgr2=0x7ff458004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff473bad640 1 --2- 172.21.15.117:0/3126884473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff458003f90 0x7ff458004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff45c001710 tx=0x7ff458005830 comp rx=0 tx=0).stop 2024-09-17T05:27:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff47214a640 1 -- 172.21.15.117:0/3126884473 reap_dead start 2024-09-17T05:27:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff473bad640 1 -- 172.21.15.117:0/3126884473 shutdown_connections 2024-09-17T05:27:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff473bad640 1 -- 172.21.15.117:0/3126884473 >> 172.21.15.117:0/3126884473 conn(0x7ff46c072e30 msgr2=0x7ff46c073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:20.320 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff473bad640 1 -- 172.21.15.117:0/3126884473 shutdown_connections 2024-09-17T05:27:20.320 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.317+0000 7ff473bad640 1 -- 172.21.15.117:0/3126884473 wait complete. 2024-09-17T05:27:20.320 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.319+0000 7ff473bad640 1 Processor -- start 2024-09-17T05:27:20.321 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.319+0000 7ff473bad640 1 -- start start 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff473bad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff458003f90 0x7ff46c0c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff473bad640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c0c5a90 0x7ff46c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff473bad640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 0x7ff46c147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff473bad640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c0c5910 con 0x7ff46c147060 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff473bad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c0c5790 con 0x7ff458003f90 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff473bad640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff46c1533d0 con 0x7ff46c0c5a90 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.320+0000 7ff47214a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 0x7ff46c147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.323 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 0x7ff46c147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45598/0 (socket says 172.21.15.117:45598) 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 -- 172.21.15.117:0/282949139 learned_addr learned my addr 172.21.15.117:0/282949139 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff471949640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff458003f90 0x7ff46c0c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff471148640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c0c5a90 0x7ff46c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 -- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c0c5a90 msgr2=0x7ff46c0b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c0c5a90 0x7ff46c0b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 -- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff458003f90 msgr2=0x7ff46c0c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff458003f90 0x7ff46c0c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 -- 172.21.15.117:0/282949139 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c15c280 con 0x7ff46c147060 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff47214a640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 0x7ff46c147470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff464005fd0 tx=0x7ff46400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464004b20 con 0x7ff46c147060 2024-09-17T05:27:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff464004cc0 con 0x7ff46c147060 2024-09-17T05:27:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464003bd0 con 0x7ff46c147060 2024-09-17T05:27:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46c149a70 con 0x7ff46c147060 2024-09-17T05:27:20.325 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.321+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46c156940 con 0x7ff46c147060 2024-09-17T05:27:20.328 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.322+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff46c0c3720 con 0x7ff46c147060 2024-09-17T05:27:20.328 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.323+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff464004030 con 0x7ff46c147060 2024-09-17T05:27:20.328 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.323+0000 7ff4627fc640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff450078bc0 0x7ff45007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.328 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.323+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff46409ae40 con 0x7ff46c147060 2024-09-17T05:27:20.328 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.324+0000 7ff471949640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff450078bc0 0x7ff45007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.328 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.324+0000 7ff471949640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff450078bc0 0x7ff45007b0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff468006b70 tx=0x7ff468006720 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.328+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff464063010 con 0x7ff46c147060 2024-09-17T05:27:20.504 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.502+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff46c0caa90 con 0x7ff46c147060 2024-09-17T05:27:20.504 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.503+0000 7ff4627fc640 1 -- 172.21.15.117:0/282949139 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12968 (secure 0 0 0) 0x7ff464067a40 con 0x7ff46c147060 2024-09-17T05:27:20.505 INFO:teuthology.misc.health.smithi117.stdout: 2024-09-17T05:27:20.505 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":17,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:16.134135+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff450078bc0 msgr2=0x7ff45007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff450078bc0 0x7ff45007b0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff468006b70 tx=0x7ff468006720 comp rx=0 tx=0).stop 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 msgr2=0x7ff46c147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 0x7ff46c147470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff464005fd0 tx=0x7ff46400f3c0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 shutdown_connections 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff450078bc0 0x7ff45007b0d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.507 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c147060 0x7ff46c147470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.508 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c0c5a90 0x7ff46c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.508 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 --2- 172.21.15.117:0/282949139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff458003f90 0x7ff46c0c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.508 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 >> 172.21.15.117:0/282949139 conn(0x7ff46c072e30 msgr2=0x7ff46c1579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:20.508 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 shutdown_connections 2024-09-17T05:27:20.508 INFO:teuthology.misc.health.smithi117.stderr:2024-09-17T05:27:20.505+0000 7ff473bad640 1 -- 172.21.15.117:0/282949139 wait complete. 2024-09-17T05:27:20.517 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-17T05:27:20.517 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T05:27:20.517 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:27:20.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:27:20.520 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:27:20.520 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:27:20.520 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:27:20.522 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:27:20.522 INFO:tasks.ceph.mds.e:Restarting daemon 2024-09-17T05:27:20.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-09-17T05:27:20.525 INFO:tasks.ceph.mds.e:Started 2024-09-17T05:27:20.526 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T05:27:20.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T05:27:20.530 INFO:tasks.ceph.mds.b:Started 2024-09-17T05:27:20.530 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:27:20.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:27:20.534 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:27:20.534 INFO:tasks.ceph.mds.f:Restarting daemon 2024-09-17T05:27:20.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-09-17T05:27:20.537 INFO:tasks.ceph.mds.f:Started 2024-09-17T05:27:20.537 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-17T05:27:20.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:20.609 INFO:tasks.ceph.mds.a.smithi117.stdout:starting mds.a at 2024-09-17T05:27:20.609 INFO:tasks.ceph.mds.c.smithi117.stdout:starting mds.c at 2024-09-17T05:27:20.611 INFO:tasks.ceph.mds.e.smithi117.stdout:starting mds.e at 2024-09-17T05:27:20.630 INFO:tasks.ceph.mds.b.smithi119.stdout:starting mds.b at 2024-09-17T05:27:20.632 INFO:tasks.ceph.mds.d.smithi119.stdout:starting mds.d at 2024-09-17T05:27:20.634 INFO:tasks.ceph.mds.f.smithi119.stdout:starting mds.f at 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 -- 172.21.15.117:0/2511958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7450153260 msgr2=0x7f74500c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 --2- 172.21.15.117:0/2511958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7450153260 0x7f74500c37c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7440005fd0 tx=0x7f744002a670 comp rx=0 tx=0).stop 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 -- 172.21.15.117:0/2511958 shutdown_connections 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 --2- 172.21.15.117:0/2511958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7450153260 0x7f74500c37c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 -- 172.21.15.117:0/2511958 >> 172.21.15.117:0/2511958 conn(0x7f7450076430 msgr2=0x7f7450072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 -- 172.21.15.117:0/2511958 shutdown_connections 2024-09-17T05:27:20.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.658+0000 7f74580e5640 1 -- 172.21.15.117:0/2511958 wait complete. 2024-09-17T05:27:20.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.660+0000 7f74580e5640 1 Processor -- start 2024-09-17T05:27:20.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.660+0000 7f74580e5640 1 -- start start 2024-09-17T05:27:20.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f74580e5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 0x7f745014c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f74580e5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7450153260 0x7f74500b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f74580e5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f745015d220 0x7f745014f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f74580e5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f74500cbc30 con 0x7f74500cad30 2024-09-17T05:27:20.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f74580e5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f745015cd00 con 0x7f745015d220 2024-09-17T05:27:20.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f74580e5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f74500c9e40 con 0x7f7450153260 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f7455e81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 0x7f745014c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f7455e81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 0x7f745014c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45648/0 (socket says 172.21.15.117:45648) 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f7455e81640 1 -- 172.21.15.117:0/2840519029 learned_addr learned my addr 172.21.15.117:0/2840519029 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f7455680640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7450153260 0x7f74500b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.661+0000 7f7456682640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f745015d220 0x7f745014f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f7455e81640 1 -- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7450153260 msgr2=0x7f74500b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f7455e81640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7450153260 0x7f74500b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f7455e81640 1 -- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f745015d220 msgr2=0x7f745014f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f7455e81640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f745015d220 0x7f745014f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f7455e81640 1 -- 172.21.15.117:0/2840519029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74500b3c30 con 0x7f74500cad30 2024-09-17T05:27:20.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f7455e81640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 0x7f745014c640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f744c00c370 tx=0x7f744c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c017070 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f744c0035f0 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c003a40 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74501ef560 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.662+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74501efb20 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.663+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74500ca900 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.664+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f744c01a070 con 0x7f74500cad30 2024-09-17T05:27:20.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.665+0000 7f74467fc640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7434078b90 0x7f743407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:20.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.665+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f744c099ff0 con 0x7f74500cad30 2024-09-17T05:27:20.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.665+0000 7f7455680640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7434078b90 0x7f743407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:20.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.665+0000 7f7455680640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7434078b90 0x7f743407b0a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7440033000 tx=0x7f7440004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:20.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.668+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f744c021020 con 0x7f74500cad30 2024-09-17T05:27:20.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.876+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74500ca300 con 0x7f74500cad30 2024-09-17T05:27:20.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.877+0000 7f74467fc640 1 -- 172.21.15.117:0/2840519029 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1437 (secure 0 0 0) 0x7f744c0621a0 con 0x7f74500cad30 2024-09-17T05:27:20.879 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:20.879 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":2,"btime":"2024-09-17T05:27:20:613619+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-17T05:27:20.879 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 2 2024-09-17T05:27:20.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7434078b90 msgr2=0x7f743407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7434078b90 0x7f743407b0a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7440033000 tx=0x7f7440004730 comp rx=0 tx=0).stop 2024-09-17T05:27:20.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 msgr2=0x7f745014c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:20.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 0x7f745014c640 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f744c00c370 tx=0x7f744c00c830 comp rx=0 tx=0).stop 2024-09-17T05:27:20.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 shutdown_connections 2024-09-17T05:27:20.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7434078b90 0x7f743407b0a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f745015d220 0x7f745014f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7450153260 0x7f74500b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 --2- 172.21.15.117:0/2840519029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74500cad30 0x7f745014c640 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:20.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 >> 172.21.15.117:0/2840519029 conn(0x7f7450076430 msgr2=0x7f745014eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:20.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 shutdown_connections 2024-09-17T05:27:20.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:20.879+0000 7f74580e5640 1 -- 172.21.15.117:0/2840519029 wait complete. 2024-09-17T05:27:22.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:27:22.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:27:22.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 -- 172.21.15.117:0/1357555763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a80cb390 msgr2=0x7f65a80d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 --2- 172.21.15.117:0/1357555763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a80cb390 0x7f65a80d08f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6598009830 tx=0x7f659802a5a0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 -- 172.21.15.117:0/1357555763 shutdown_connections 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 --2- 172.21.15.117:0/1357555763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f65a80c69d0 0x7f65a80d0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 --2- 172.21.15.117:0/1357555763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a80cb390 0x7f65a80d08f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 -- 172.21.15.117:0/1357555763 >> 172.21.15.117:0/1357555763 conn(0x7f65a8075e00 msgr2=0x7f65a806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 -- 172.21.15.117:0/1357555763 shutdown_connections 2024-09-17T05:27:23.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.018+0000 7f65ae664640 1 -- 172.21.15.117:0/1357555763 wait complete. 2024-09-17T05:27:23.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.020+0000 7f65ae664640 1 Processor -- start 2024-09-17T05:27:23.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.020+0000 7f65ae664640 1 -- start start 2024-09-17T05:27:23.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65ae664640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65a80c69d0 0x7f65a80b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65ae664640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f65a80cb390 0x7f65a814e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65ae664640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 0x7f65a814e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65ae664640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f65a80c5da0 con 0x7f65a815b2f0 2024-09-17T05:27:23.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65ae664640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f65a815f6b0 con 0x7f65a80c69d0 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65ae664640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f65a81558b0 con 0x7f65a80cb390 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65acc01640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 0x7f65a814e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65acc01640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 0x7f65a814e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45656/0 (socket says 172.21.15.117:45656) 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65acc01640 1 -- 172.21.15.117:0/2551486577 learned_addr learned my addr 172.21.15.117:0/2551486577 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65a7fff640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65a80c69d0 0x7f65a80b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.021+0000 7f65a77fe640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f65a80cb390 0x7f65a814e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65acc01640 1 -- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f65a80cb390 msgr2=0x7f65a814e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65acc01640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f65a80cb390 0x7f65a814e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65acc01640 1 -- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65a80c69d0 msgr2=0x7f65a80b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65acc01640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65a80c69d0 0x7f65a80b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65acc01640 1 -- 172.21.15.117:0/2551486577 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a81567d0 con 0x7f65a815b2f0 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65acc01640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 0x7f65a814e7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f65a000e960 tx=0x7f65a000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:23.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0004d20 con 0x7f65a815b2f0 2024-09-17T05:27:23.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65a000cac0 con 0x7f65a815b2f0 2024-09-17T05:27:23.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a8156a90 con 0x7f65a815b2f0 2024-09-17T05:27:23.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a0005020 con 0x7f65a815b2f0 2024-09-17T05:27:23.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.022+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a81e6890 con 0x7f65a815b2f0 2024-09-17T05:27:23.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.023+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a80c7950 con 0x7f65a815b2f0 2024-09-17T05:27:23.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.024+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f65a00051c0 con 0x7f65a815b2f0 2024-09-17T05:27:23.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.024+0000 7f65a4ff9640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6594078b90 0x7f659407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.024+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f65a009b1f0 con 0x7f65a815b2f0 2024-09-17T05:27:23.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.024+0000 7f65a7fff640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6594078b90 0x7f659407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.025+0000 7f65a7fff640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6594078b90 0x7f659407b0a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6598034000 tx=0x7f6598004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:23.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.029+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f65a0016030 con 0x7f65a815b2f0 2024-09-17T05:27:23.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.237+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65a80cb810 con 0x7f65a815b2f0 2024-09-17T05:27:23.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.237+0000 7f65a4ff9640 1 -- 172.21.15.117:0/2551486577 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5566 (secure 0 0 0) 0x7f65a0063330 con 0x7f65a815b2f0 2024-09-17T05:27:23.239 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:23.239 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":3,"btime":"2024-09-17T05:27:21:615102+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4434,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4440,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-17T05:27:23.239 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 3 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6594078b90 msgr2=0x7f659407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6594078b90 0x7f659407b0a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6598034000 tx=0x7f6598004730 comp rx=0 tx=0).stop 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 msgr2=0x7f65a814e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 0x7f65a814e7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f65a000e960 tx=0x7f65a000ee20 comp rx=0 tx=0).stop 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 shutdown_connections 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6594078b90 0x7f659407b0a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65a815b2f0 0x7f65a814e7a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f65a80cb390 0x7f65a814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 --2- 172.21.15.117:0/2551486577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65a80c69d0 0x7f65a80b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 >> 172.21.15.117:0/2551486577 conn(0x7f65a8075e00 msgr2=0x7f65a8158f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:23.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 shutdown_connections 2024-09-17T05:27:23.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.239+0000 7f65ae664640 1 -- 172.21.15.117:0/2551486577 wait complete. 2024-09-17T05:27:23.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T05:27:23.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 -- 172.21.15.117:0/1165369038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c0b56c0 msgr2=0x7f845c166c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 --2- 172.21.15.117:0/1165369038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c0b56c0 0x7f845c166c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8454005fd0 tx=0x7f845402a9d0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 -- 172.21.15.117:0/1165369038 shutdown_connections 2024-09-17T05:27:23.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 --2- 172.21.15.117:0/1165369038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c0b56c0 0x7f845c166c80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 -- 172.21.15.117:0/1165369038 >> 172.21.15.117:0/1165369038 conn(0x7f845c074d80 msgr2=0x7f845c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:23.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 -- 172.21.15.117:0/1165369038 shutdown_connections 2024-09-17T05:27:23.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.363+0000 7f846224d640 1 -- 172.21.15.117:0/1165369038 wait complete. 2024-09-17T05:27:23.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.364+0000 7f846224d640 1 Processor -- start 2024-09-17T05:27:23.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.364+0000 7f846224d640 1 -- start start 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f845c0b52b0 0x7f845c1f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f845c0b56c0 0x7f845c15da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 0x7f845c152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f845c1558b0 con 0x7f845c15bd70 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f845c0cd0d0 con 0x7f845c0b52b0 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f845c0cb290 con 0x7f845c0b56c0 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 0x7f845c152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 0x7f845c152ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45674/0 (socket says 172.21.15.117:45674) 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 -- 172.21.15.117:0/4129386176 learned_addr learned my addr 172.21.15.117:0/4129386176 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845affd640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f845c0b56c0 0x7f845c15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845b7fe640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f845c0b52b0 0x7f845c1f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 -- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f845c0b56c0 msgr2=0x7f845c15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f845c0b56c0 0x7f845c15da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 -- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f845c0b52b0 msgr2=0x7f845c1f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f845c0b52b0 0x7f845c1f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 -- 172.21.15.117:0/4129386176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845c1f4f90 con 0x7f845c15bd70 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845affd640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f845c0b56c0 0x7f845c15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845b7fe640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f845c0b52b0 0x7f845c1f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f845bfff640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 0x7f845c152ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8454009cd0 tx=0x7f8454003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454045070 con 0x7f845c15bd70 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8454003cf0 con 0x7f845c15bd70 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845c158680 con 0x7f845c15bd70 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.366+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845402fcc0 con 0x7f845c15bd70 2024-09-17T05:27:23.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.367+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845c15e520 con 0x7f845c15bd70 2024-09-17T05:27:23.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.368+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f845c0d0870 con 0x7f845c15bd70 2024-09-17T05:27:23.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.368+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8454048070 con 0x7f845c15bd70 2024-09-17T05:27:23.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.369+0000 7f8443fff640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8448078bc0 0x7f844807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.369+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f84540bb370 con 0x7f845c15bd70 2024-09-17T05:27:23.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.369+0000 7f845b7fe640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8448078bc0 0x7f844807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.369+0000 7f845b7fe640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8448078bc0 0x7f844807b0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f844c007fd0 tx=0x7f844c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:23.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.373+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8454033020 con 0x7f845c15bd70 2024-09-17T05:27:23.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.550+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f845c0ca810 con 0x7f845c15bd70 2024-09-17T05:27:23.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.550+0000 7f8443fff640 1 -- 172.21.15.117:0/4129386176 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f8454083b90 con 0x7f845c15bd70 2024-09-17T05:27:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:23.552 INFO:teuthology.orchestra.run.smithi117.stdout:[ 2024-09-17T05:27:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-09-17T05:27:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "poolnum": 1, 2024-09-17T05:27:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "poolname": ".mgr" 2024-09-17T05:27:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-17T05:27:23.553 INFO:teuthology.orchestra.run.smithi117.stdout:] 2024-09-17T05:27:23.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.552+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8448078bc0 msgr2=0x7f844807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.552+0000 7f846224d640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8448078bc0 0x7f844807b0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f844c007fd0 tx=0x7f844c009040 comp rx=0 tx=0).stop 2024-09-17T05:27:23.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.552+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 msgr2=0x7f845c152ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.552+0000 7f846224d640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 0x7f845c152ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8454009cd0 tx=0x7f8454003970 comp rx=0 tx=0).stop 2024-09-17T05:27:23.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 shutdown_connections 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8448078bc0 0x7f844807b0d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f845c15bd70 0x7f845c152ee0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f845c0b56c0 0x7f845c15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 --2- 172.21.15.117:0/4129386176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f845c0b52b0 0x7f845c1f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 >> 172.21.15.117:0/4129386176 conn(0x7f845c074d80 msgr2=0x7f845c154e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 shutdown_connections 2024-09-17T05:27:23.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.553+0000 7f846224d640 1 -- 172.21.15.117:0/4129386176 wait complete. 2024-09-17T05:27:23.565 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T05:27:23.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 -- 172.21.15.117:0/3299101133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100cbdf0 msgr2=0x7f19100b43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 --2- 172.21.15.117:0/3299101133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100cbdf0 0x7f19100b43d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1900009830 tx=0x7f190002a510 comp rx=0 tx=0).stop 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 -- 172.21.15.117:0/3299101133 shutdown_connections 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 --2- 172.21.15.117:0/3299101133 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100cabc0 0x7f19100cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 --2- 172.21.15.117:0/3299101133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100cbdf0 0x7f19100b43d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 -- 172.21.15.117:0/3299101133 >> 172.21.15.117:0/3299101133 conn(0x7f1910075e00 msgr2=0x7f1910076210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 -- 172.21.15.117:0/3299101133 shutdown_connections 2024-09-17T05:27:23.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.681+0000 7f1914cbd640 1 -- 172.21.15.117:0/3299101133 wait complete. 2024-09-17T05:27:23.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.683+0000 7f1914cbd640 1 Processor -- start 2024-09-17T05:27:23.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.683+0000 7f1914cbd640 1 -- start start 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f1914cbd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 0x7f19100c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f1914cbd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100cbdf0 0x7f1910157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f1914cbd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100d0ca0 0x7f1910158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f1914cbd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f191015f090 con 0x7f19100cabc0 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f1914cbd640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f191015bfd0 con 0x7f19100d0ca0 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f1914cbd640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f191015f210 con 0x7f19100cbdf0 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f190e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 0x7f19100c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f190e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 0x7f19100c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45690/0 (socket says 172.21.15.117:45690) 2024-09-17T05:27:23.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.684+0000 7f190e59c640 1 -- 172.21.15.117:0/2390205440 learned_addr learned my addr 172.21.15.117:0/2390205440 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190ed9d640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100d0ca0 0x7f1910158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190dd9b640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100cbdf0 0x7f1910157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190e59c640 1 -- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100cbdf0 msgr2=0x7f1910157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190e59c640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100cbdf0 0x7f1910157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190e59c640 1 -- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100d0ca0 msgr2=0x7f1910158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190e59c640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100d0ca0 0x7f1910158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190e59c640 1 -- 172.21.15.117:0/2390205440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1910158740 con 0x7f19100cabc0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f190e59c640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 0x7f19100c5ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1900009950 tx=0x7f1900033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900031d60 con 0x7f19100cabc0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1900033e10 con 0x7f19100cabc0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19100c6010 con 0x7f19100cabc0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900005260 con 0x7f19100cabc0 2024-09-17T05:27:23.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.685+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19100ad090 con 0x7f19100cabc0 2024-09-17T05:27:23.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.686+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19100b5c10 con 0x7f19100cabc0 2024-09-17T05:27:23.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.687+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1900005400 con 0x7f19100cabc0 2024-09-17T05:27:23.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.687+0000 7f18feffd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f18f0078b90 0x7f18f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:23.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.687+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f19000bbae0 con 0x7f19100cabc0 2024-09-17T05:27:23.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.687+0000 7f190dd9b640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f18f0078b90 0x7f18f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:23.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.688+0000 7f190dd9b640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f18f0078b90 0x7f18f007b0a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f19100c8d80 tx=0x7f1904005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:23.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.692+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f19100b5c10 con 0x7f19100cabc0 2024-09-17T05:27:23.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:23.864+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f19100cc250 con 0x7f19100cabc0 2024-09-17T05:27:24.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.628+0000 7f18feffd640 1 -- 172.21.15.117:0/2390205440 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f190003f030 con 0x7f19100cabc0 2024-09-17T05:27:24.633 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_metadata' created 2024-09-17T05:27:24.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f18f0078b90 msgr2=0x7f18f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f18f0078b90 0x7f18f007b0a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f19100c8d80 tx=0x7f1904005f90 comp rx=0 tx=0).stop 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 msgr2=0x7f19100c5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 0x7f19100c5ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1900009950 tx=0x7f1900033c20 comp rx=0 tx=0).stop 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 shutdown_connections 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f18f0078b90 0x7f18f007b0a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19100d0ca0 0x7f1910158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100cbdf0 0x7f1910157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 --2- 172.21.15.117:0/2390205440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19100cabc0 0x7f19100c5ad0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 >> 172.21.15.117:0/2390205440 conn(0x7f1910075e00 msgr2=0x7f1910158c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.631+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 shutdown_connections 2024-09-17T05:27:24.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.632+0000 7f1914cbd640 1 -- 172.21.15.117:0/2390205440 wait complete. 2024-09-17T05:27:24.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-17T05:27:24.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 -- 172.21.15.117:0/3724118425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3800b4db0 msgr2=0x7fe3800cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:24.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 --2- 172.21.15.117:0/3724118425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3800b4db0 0x7fe3800cd290 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe378005fd0 tx=0x7fe37802a940 comp rx=0 tx=0).stop 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 -- 172.21.15.117:0/3724118425 shutdown_connections 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 --2- 172.21.15.117:0/3724118425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3800b4db0 0x7fe3800cd290 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 --2- 172.21.15.117:0/3724118425 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe380151ec0 0x7fe3800b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 -- 172.21.15.117:0/3724118425 >> 172.21.15.117:0/3724118425 conn(0x7fe38006e170 msgr2=0x7fe38006e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 -- 172.21.15.117:0/3724118425 shutdown_connections 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.774+0000 7fe387d62640 1 -- 172.21.15.117:0/3724118425 wait complete. 2024-09-17T05:27:24.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.775+0000 7fe387d62640 1 Processor -- start 2024-09-17T05:27:24.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.775+0000 7fe387d62640 1 -- start start 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe3800b4db0 0x7fe3800d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 0x7fe3801f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe380151ec0 0x7fe3801e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe38015f090 con 0x7fe3800cbbd0 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe38015edb0 con 0x7fe380151ec0 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe3800b43e0 con 0x7fe3800b4db0 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 0x7fe3801f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 0x7fe3801f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40894/0 (socket says 172.21.15.117:40894) 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 -- 172.21.15.117:0/1199869801 learned_addr learned my addr 172.21.15.117:0/1199869801 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3862ff640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe380151ec0 0x7fe3801e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:24.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe385afe640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe3800b4db0 0x7fe3800d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 -- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe3800b4db0 msgr2=0x7fe3800d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe3800b4db0 0x7fe3800d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 -- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe380151ec0 msgr2=0x7fe3801e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe380151ec0 0x7fe3801e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 -- 172.21.15.117:0/1199869801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3801f27f0 con 0x7fe3800cbbd0 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3862ff640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe380151ec0 0x7fe3801e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe385afe640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe3800b4db0 0x7fe3800d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3852fd640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 0x7fe3801f22b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe37000ea20 tx=0x7fe37000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37000cd50 con 0x7fe3800cbbd0 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3801e6c30 con 0x7fe3800cbbd0 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe370004020 con 0x7fe3800cbbd0 2024-09-17T05:27:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370004470 con 0x7fe3800cbbd0 2024-09-17T05:27:24.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.777+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3800ceae0 con 0x7fe3800cbbd0 2024-09-17T05:27:24.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.778+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3800b4870 con 0x7fe3800cbbd0 2024-09-17T05:27:24.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.779+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe3700035f0 con 0x7fe3800cbbd0 2024-09-17T05:27:24.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.780+0000 7fe3767fc640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe364078bc0 0x7fe36407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:24.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.780+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fe37009b650 con 0x7fe3800cbbd0 2024-09-17T05:27:24.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.780+0000 7fe385afe640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe364078bc0 0x7fe36407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:24.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.781+0000 7fe385afe640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe364078bc0 0x7fe36407b0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe37c007fd0 tx=0x7fe37c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:24.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.784+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe370063d00 con 0x7fe3800cbbd0 2024-09-17T05:27:24.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:24.958+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fe3801557e0 con 0x7fe3800cbbd0 2024-09-17T05:27:25.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.633+0000 7fe3767fc640 1 -- 172.21.15.117:0/1199869801 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7fe370063600 con 0x7fe3800cbbd0 2024-09-17T05:27:25.634 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data' created 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe364078bc0 msgr2=0x7fe36407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe364078bc0 0x7fe36407b0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe37c007fd0 tx=0x7fe37c009040 comp rx=0 tx=0).stop 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 msgr2=0x7fe3801f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 0x7fe3801f22b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe37000ea20 tx=0x7fe37000eee0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 shutdown_connections 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe364078bc0 0x7fe36407b0d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe380151ec0 0x7fe3801e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3800cbbd0 0x7fe3801f22b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 --2- 172.21.15.117:0/1199869801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe3800b4db0 0x7fe3800d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.636+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 >> 172.21.15.117:0/1199869801 conn(0x7fe38006e170 msgr2=0x7fe380159900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:25.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.637+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 shutdown_connections 2024-09-17T05:27:25.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.637+0000 7fe387d62640 1 -- 172.21.15.117:0/1199869801 wait complete. 2024-09-17T05:27:25.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T05:27:25.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 -- 172.21.15.117:0/1773138304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a40c67a0 msgr2=0x7f41a4152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/1773138304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a40c67a0 0x7f41a4152210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4194005fd0 tx=0x7f419402a4a0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 -- 172.21.15.117:0/1773138304 shutdown_connections 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/1773138304 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 0x7f41a4071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/1773138304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a40c67a0 0x7f41a4152210 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 -- 172.21.15.117:0/1773138304 >> 172.21.15.117:0/1773138304 conn(0x7f41a406e590 msgr2=0x7f41a40cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 -- 172.21.15.117:0/1773138304 shutdown_connections 2024-09-17T05:27:25.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.778+0000 7f41a8f8a640 1 -- 172.21.15.117:0/1773138304 wait complete. 2024-09-17T05:27:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.780+0000 7f41a8f8a640 1 Processor -- start 2024-09-17T05:27:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.780+0000 7f41a8f8a640 1 -- start start 2024-09-17T05:27:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a8f8a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 0x7f41a41f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:25.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a8f8a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 0x7f41a4159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a8f8a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41a40c67a0 0x7f41a414e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a8f8a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40c7ff0 con 0x7f41a4071850 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a8f8a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40ce0d0 con 0x7f41a40c67a0 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a8f8a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f41a40c83a0 con 0x7f41a40c5660 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 0x7f41a41f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 0x7f41a41f5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40910/0 (socket says 172.21.15.117:40910) 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a259c640 1 -- 172.21.15.117:0/4250007862 learned_addr learned my addr 172.21.15.117:0/4250007862 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f4199d9b640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 0x7f41a4159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.781+0000 7f41a2d9d640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41a40c67a0 0x7f41a414e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a259c640 1 -- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 msgr2=0x7f41a4159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a259c640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 0x7f41a4159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a259c640 1 -- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41a40c67a0 msgr2=0x7f41a414e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a259c640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41a40c67a0 0x7f41a414e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a259c640 1 -- 172.21.15.117:0/4250007862 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a415a390 con 0x7f41a4071850 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a2d9d640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41a40c67a0 0x7f41a414e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f4199d9b640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 0x7f41a4159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a259c640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 0x7f41a41f5990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4194009850 tx=0x7f41940039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419403e030 con 0x7f41a4071850 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41940047a0 con 0x7f41a4071850 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a41f4ed0 con 0x7f41a4071850 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194035040 con 0x7f41a4071850 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.782+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a40cd150 con 0x7f41a4071850 2024-09-17T05:27:25.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.783+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a40c7940 con 0x7f41a4071850 2024-09-17T05:27:25.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.784+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4194041030 con 0x7f41a4071850 2024-09-17T05:27:25.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.784+0000 7f419b7fe640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4184078bc0 0x7f418407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:25.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.784+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f41940bb640 con 0x7f41a4071850 2024-09-17T05:27:25.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.785+0000 7f4199d9b640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4184078bc0 0x7f418407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:25.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.785+0000 7f4199d9b640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4184078bc0 0x7f418407b0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f418c006fd0 tx=0x7f418c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:25.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:25.789+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4194083ae0 con 0x7f41a4071850 2024-09-17T05:27:26.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.001+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f41a40c8d00 con 0x7f41a4071850 2024-09-17T05:27:26.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.635+0000 7f419b7fe640 1 -- 172.21.15.117:0/4250007862 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f41940833e0 con 0x7f41a4071850 2024-09-17T05:27:26.636 INFO:teuthology.orchestra.run.smithi117.stderr:new fs with metadata pool 2 and data pool 3 2024-09-17T05:27:26.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.639+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4184078bc0 msgr2=0x7f418407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:26.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.639+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4184078bc0 0x7f418407b0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f418c006fd0 tx=0x7f418c008040 comp rx=0 tx=0).stop 2024-09-17T05:27:26.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.639+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 msgr2=0x7f41a41f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:26.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.639+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 0x7f41a41f5990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4194009850 tx=0x7f41940039e0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.640+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 shutdown_connections 2024-09-17T05:27:26.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.640+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4184078bc0 0x7f418407b0d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.640+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41a40c67a0 0x7f41a414e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.640+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41a40c5660 0x7f41a4159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.640+0000 7f41a8f8a640 1 --2- 172.21.15.117:0/4250007862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41a4071850 0x7f41a41f5990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.640+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 >> 172.21.15.117:0/4250007862 conn(0x7f41a406e590 msgr2=0x7f41a41575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:26.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.642+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 shutdown_connections 2024-09-17T05:27:26.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.643+0000 7f41a8f8a640 1 -- 172.21.15.117:0/4250007862 wait complete. 2024-09-17T05:27:26.676 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-17T05:27:26.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-17T05:27:26.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 -- 172.21.15.117:0/4165638983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d015ba60 msgr2=0x7f20d0165bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:26.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 --2- 172.21.15.117:0/4165638983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d015ba60 0x7f20d0165bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f20c8005fd0 tx=0x7f20c802a670 comp rx=0 tx=0).stop 2024-09-17T05:27:26.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 -- 172.21.15.117:0/4165638983 shutdown_connections 2024-09-17T05:27:26.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 --2- 172.21.15.117:0/4165638983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d015ba60 0x7f20d0165bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 --2- 172.21.15.117:0/4165638983 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20d01555b0 0x7f20d015b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 -- 172.21.15.117:0/4165638983 >> 172.21.15.117:0/4165638983 conn(0x7f20d00744d0 msgr2=0x7f20d0074910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:26.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 -- 172.21.15.117:0/4165638983 shutdown_connections 2024-09-17T05:27:26.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.796+0000 7f20d8313640 1 -- 172.21.15.117:0/4165638983 wait complete. 2024-09-17T05:27:26.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.798+0000 7f20d8313640 1 Processor -- start 2024-09-17T05:27:26.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.798+0000 7f20d8313640 1 -- start start 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d8313640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d00c6b70 0x7f20d01571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d8313640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 0x7f20d006d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d8313640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20d015ba60 0x7f20d006d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d8313640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f20d00ccfc0 con 0x7f20d01555b0 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d8313640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f20d00cb6f0 con 0x7f20d00c6b70 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d8313640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f20d00cc460 con 0x7f20d015ba60 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d58ae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 0x7f20d006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:26.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d58ae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 0x7f20d006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40924/0 (socket says 172.21.15.117:40924) 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.799+0000 7f20d58ae640 1 -- 172.21.15.117:0/1783789424 learned_addr learned my addr 172.21.15.117:0/1783789424 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d60af640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d00c6b70 0x7f20d01571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d68b0640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20d015ba60 0x7f20d006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d58ae640 1 -- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20d015ba60 msgr2=0x7f20d006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d58ae640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20d015ba60 0x7f20d006d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d58ae640 1 -- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d00c6b70 msgr2=0x7f20d01571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d58ae640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d00c6b70 0x7f20d01571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d58ae640 1 -- 172.21.15.117:0/1783789424 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d01f31c0 con 0x7f20d01555b0 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d58ae640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 0x7f20d006d050 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20c400e700 tx=0x7f20c400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:26.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c400ca50 con 0x7f20d01555b0 2024-09-17T05:27:26.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20c400cce0 con 0x7f20d01555b0 2024-09-17T05:27:26.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4010690 con 0x7f20d01555b0 2024-09-17T05:27:26.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d01f2400 con 0x7f20d01555b0 2024-09-17T05:27:26.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.800+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d01f29c0 con 0x7f20d01555b0 2024-09-17T05:27:26.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.801+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d00c70a0 con 0x7f20d01555b0 2024-09-17T05:27:26.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.802+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f20c4010830 con 0x7f20d01555b0 2024-09-17T05:27:26.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.802+0000 7f20ceffd640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f20bc078b90 0x7f20bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:26.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.802+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f20c409b620 con 0x7f20d01555b0 2024-09-17T05:27:26.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.803+0000 7f20d60af640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f20bc078b90 0x7f20bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:26.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.803+0000 7f20d60af640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f20bc078b90 0x7f20bc07b0a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f20c0001e20 tx=0x7f20c0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:26.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.807+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f20d00c70a0 con 0x7f20d01555b0 2024-09-17T05:27:26.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:26.990+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f20d01595a0 con 0x7f20d01555b0 2024-09-17T05:27:27.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.650+0000 7f20ceffd640 1 -- 172.21.15.117:0/1783789424 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f20d01595a0 con 0x7f20d01555b0 2024-09-17T05:27:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.653+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f20bc078b90 msgr2=0x7f20bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.653+0000 7f20d8313640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f20bc078b90 0x7f20bc07b0a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f20c0001e20 tx=0x7f20c0009040 comp rx=0 tx=0).stop 2024-09-17T05:27:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 msgr2=0x7f20d006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 0x7f20d006d050 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20c400e700 tx=0x7f20c400ebc0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 shutdown_connections 2024-09-17T05:27:27.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f20bc078b90 0x7f20bc07b0a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20d015ba60 0x7f20d006d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d01555b0 0x7f20d006d050 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 --2- 172.21.15.117:0/1783789424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d00c6b70 0x7f20d01571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 >> 172.21.15.117:0/1783789424 conn(0x7f20d00744d0 msgr2=0x7f20d00b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:27.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 shutdown_connections 2024-09-17T05:27:27.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.654+0000 7f20d8313640 1 -- 172.21.15.117:0/1783789424 wait complete. 2024-09-17T05:27:27.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 -- 172.21.15.117:0/711831321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0741d0 msgr2=0x7f730c161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 --2- 172.21.15.117:0/711831321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0741d0 0x7f730c161f10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f730c0cd5d0 tx=0x7f730402a940 comp rx=0 tx=0).stop 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 -- 172.21.15.117:0/711831321 shutdown_connections 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 --2- 172.21.15.117:0/711831321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0741d0 0x7f730c161f10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 --2- 172.21.15.117:0/711831321 >> v2:172.21.15.119:3300/0 conn(0x7f730c0c9760 0x7f730c1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 -- 172.21.15.117:0/711831321 >> 172.21.15.117:0/711831321 conn(0x7f730c0ccbe0 msgr2=0x7f730c0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 -- 172.21.15.117:0/711831321 shutdown_connections 2024-09-17T05:27:27.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.792+0000 7f7310ba8640 1 -- 172.21.15.117:0/711831321 wait complete. 2024-09-17T05:27:27.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.794+0000 7f7310ba8640 1 Processor -- start 2024-09-17T05:27:27.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.794+0000 7f7310ba8640 1 -- start start 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7310ba8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f730c073dc0 0x7f730c14a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7310ba8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f730c0741d0 0x7f730c1f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7310ba8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 0x7f730c1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7310ba8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f730c0c9ba0 con 0x7f730c0c9760 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7310ba8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f730c0745e0 con 0x7f730c0741d0 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7310ba8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f730c0c9d20 con 0x7f730c073dc0 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 0x7f730c1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:27.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 0x7f730c1501d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40944/0 (socket says 172.21.15.117:40944) 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 -- 172.21.15.117:0/1830242822 learned_addr learned my addr 172.21.15.117:0/1830242822 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730a59c640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f730c073dc0 0x7f730c14a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f7309d9b640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f730c0741d0 0x7f730c1f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 -- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f730c073dc0 msgr2=0x7f730c14a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f730c073dc0 0x7f730c14a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 -- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f730c0741d0 msgr2=0x7f730c1f08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f730c0741d0 0x7f730c1f08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 -- 172.21.15.117:0/1830242822 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c1f0e00 con 0x7f730c0c9760 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.795+0000 7f730ad9d640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 0x7f730c1501d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f730402af30 tx=0x7f7304031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.796+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730403e070 con 0x7f730c0c9760 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.796+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7304004540 con 0x7f730c0c9760 2024-09-17T05:27:27.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.796+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c1571d0 con 0x7f730c0c9760 2024-09-17T05:27:27.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.796+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304030430 con 0x7f730c0c9760 2024-09-17T05:27:27.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.796+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c0cafa0 con 0x7f730c0c9760 2024-09-17T05:27:27.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.797+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c0caab0 con 0x7f730c0c9760 2024-09-17T05:27:27.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.797+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7304043460 con 0x7f730c0c9760 2024-09-17T05:27:27.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.798+0000 7f72faffd640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f72ec078b90 0x7f72ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:27.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.798+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f73040bc8c0 con 0x7f730c0c9760 2024-09-17T05:27:27.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.798+0000 7f730a59c640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f72ec078b90 0x7f72ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:27.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.799+0000 7f730a59c640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f72ec078b90 0x7f72ec07b0a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00b7b0 tx=0x7f72fc005e90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:27.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.802+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7304084cf0 con 0x7f730c0c9760 2024-09-17T05:27:27.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:27.975+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f730c0c3fb0 con 0x7f730c0c9760 2024-09-17T05:27:29.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.654+0000 7f72faffd640 1 -- 172.21.15.117:0/1830242822 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f73040845f0 con 0x7f730c0c9760 2024-09-17T05:27:29.655 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data_ec' created 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f72ec078b90 msgr2=0x7f72ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f72ec078b90 0x7f72ec07b0a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00b7b0 tx=0x7f72fc005e90 comp rx=0 tx=0).stop 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 msgr2=0x7f730c1501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 0x7f730c1501d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f730402af30 tx=0x7f7304031ea0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 shutdown_connections 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f72ec078b90 0x7f72ec07b0a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f730c0c9760 0x7f730c1501d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.658+0000 7f7310ba8640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f730c0741d0 0x7f730c1f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.659+0000 7f7310ba8640 1 --2- 172.21.15.117:0/1830242822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f730c073dc0 0x7f730c14a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.659+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 >> 172.21.15.117:0/1830242822 conn(0x7f730c0ccbe0 msgr2=0x7f730c06cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:29.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.659+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 shutdown_connections 2024-09-17T05:27:29.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.659+0000 7f7310ba8640 1 -- 172.21.15.117:0/1830242822 wait complete. 2024-09-17T05:27:29.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-17T05:27:29.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.799+0000 7f9ab7121640 1 -- 172.21.15.117:0/1063609450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0153120 msgr2=0x7f9ab0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:29.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.799+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1063609450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0153120 0x7f9ab0159100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009830 tx=0x7f9aa002a5a0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.799+0000 7f9ab7121640 1 -- 172.21.15.117:0/1063609450 shutdown_connections 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.799+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1063609450 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ab0159640 0x7f9ab0163a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.799+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1063609450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0153120 0x7f9ab0159100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.800+0000 7f9ab7121640 1 -- 172.21.15.117:0/1063609450 >> 172.21.15.117:0/1063609450 conn(0x7f9ab00722b0 msgr2=0x7f9ab0075e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.800+0000 7f9ab7121640 1 -- 172.21.15.117:0/1063609450 shutdown_connections 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.800+0000 7f9ab7121640 1 -- 172.21.15.117:0/1063609450 wait complete. 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.801+0000 7f9ab7121640 1 Processor -- start 2024-09-17T05:27:29.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.801+0000 7f9ab7121640 1 -- start start 2024-09-17T05:27:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab7121640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ab00cf250 0x7f9ab01f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab7121640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 0x7f9ab00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab7121640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0159640 0x7f9ab0143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab7121640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab00c3fb0 con 0x7f9ab0153120 2024-09-17T05:27:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab7121640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab00c3cd0 con 0x7f9ab0159640 2024-09-17T05:27:29.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab7121640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab0153530 con 0x7f9ab00cf250 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 0x7f9ab00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 0x7f9ab00b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40952/0 (socket says 172.21.15.117:40952) 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 -- 172.21.15.117:0/1290377543 learned_addr learned my addr 172.21.15.117:0/1290377543 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab56be640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0159640 0x7f9ab0143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9ab4ebd640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ab00cf250 0x7f9ab01f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 -- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ab00cf250 msgr2=0x7f9ab01f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ab00cf250 0x7f9ab01f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 -- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0159640 msgr2=0x7f9ab0143af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0159640 0x7f9ab0143af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 -- 172.21.15.117:0/1290377543 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab01f3fc0 con 0x7f9ab0153120 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.803+0000 7f9aa7fff640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 0x7f9ab00b2cf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009950 tx=0x7f9aa0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.804+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0031d60 con 0x7f9ab0153120 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.804+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab01473c0 con 0x7f9ab0153120 2024-09-17T05:27:29.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.804+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9aa0033e10 con 0x7f9ab0153120 2024-09-17T05:27:29.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.804+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0005260 con 0x7f9ab0153120 2024-09-17T05:27:29.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.804+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab0147940 con 0x7f9ab0153120 2024-09-17T05:27:29.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.805+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab00cc550 con 0x7f9ab0153120 2024-09-17T05:27:29.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.805+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9aa0005400 con 0x7f9ab0153120 2024-09-17T05:27:29.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.806+0000 7f9aa57fa640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9a9c078bc0 0x7f9a9c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:29.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.806+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f9aa00bc0b0 con 0x7f9ab0153120 2024-09-17T05:27:29.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.806+0000 7f9ab4ebd640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9a9c078bc0 0x7f9a9c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:29.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.807+0000 7f9ab4ebd640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9a9c078bc0 0x7f9a9c07b0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9aac001720 tx=0x7f9aac009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:29.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.811+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9aa003f030 con 0x7f9ab0153120 2024-09-17T05:27:29.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:29.986+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f9ab014cf40 con 0x7f9ab0153120 2024-09-17T05:27:30.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.656+0000 7f9aa57fa640 1 -- 172.21.15.117:0/1290377543 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f9aa0083c20 con 0x7f9ab0153120 2024-09-17T05:27:30.657 INFO:teuthology.orchestra.run.smithi117.stderr:set pool 4 allow_ec_overwrites to true 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.659+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9a9c078bc0 msgr2=0x7f9a9c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.659+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9a9c078bc0 0x7f9a9c07b0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9aac001720 tx=0x7f9aac009040 comp rx=0 tx=0).stop 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.659+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 msgr2=0x7f9ab00b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.659+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 0x7f9ab00b2cf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009950 tx=0x7f9aa0033c20 comp rx=0 tx=0).stop 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.660+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 shutdown_connections 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.660+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9a9c078bc0 0x7f9a9c07b0d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:30.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.660+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab0159640 0x7f9ab0143af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:30.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.660+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ab0153120 0x7f9ab00b2cf0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:30.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.660+0000 7f9ab7121640 1 --2- 172.21.15.117:0/1290377543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9ab00cf250 0x7f9ab01f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:30.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.660+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 >> 172.21.15.117:0/1290377543 conn(0x7f9ab00722b0 msgr2=0x7f9ab014d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:30.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.661+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 shutdown_connections 2024-09-17T05:27:30.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.661+0000 7f9ab7121640 1 -- 172.21.15.117:0/1290377543 wait complete. 2024-09-17T05:27:30.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd503350640 1 -- 172.21.15.117:0/2603461681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4e8004d40 msgr2=0x7fd4e8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd503350640 1 --2- 172.21.15.117:0/2603461681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4e8004d40 0x7fd4e8005190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc155cb0 tx=0x7fd4e8005830 comp rx=0 tx=0).stop 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd5018ed640 1 -- 172.21.15.117:0/2603461681 reap_dead start 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd503350640 1 -- 172.21.15.117:0/2603461681 shutdown_connections 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd503350640 1 -- 172.21.15.117:0/2603461681 >> 172.21.15.117:0/2603461681 conn(0x7fd4fc06e590 msgr2=0x7fd4fc0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd503350640 1 -- 172.21.15.117:0/2603461681 shutdown_connections 2024-09-17T05:27:30.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.802+0000 7fd503350640 1 -- 172.21.15.117:0/2603461681 wait complete. 2024-09-17T05:27:30.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.804+0000 7fd503350640 1 Processor -- start 2024-09-17T05:27:30.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.804+0000 7fd503350640 1 -- start start 2024-09-17T05:27:30.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd503350640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd4e8004d40 0x7fd4fc14a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:30.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd503350640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 0x7fd4fc0c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:30.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd503350640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0c3af0 0x7fd4fc0cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:30.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd503350640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc0c7ff0 con 0x7fd4fc149960 2024-09-17T05:27:30.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd503350640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc0ce0d0 con 0x7fd4fc0c3af0 2024-09-17T05:27:30.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd503350640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd4fc0c83a0 con 0x7fd4e8004d40 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd5008eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 0x7fd4fc0c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd5008eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 0x7fd4fc0c35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40976/0 (socket says 172.21.15.117:40976) 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.805+0000 7fd5008eb640 1 -- 172.21.15.117:0/2206779656 learned_addr learned my addr 172.21.15.117:0/2206779656 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5010ec640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd4e8004d40 0x7fd4fc14a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5018ed640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0c3af0 0x7fd4fc0cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5008eb640 1 -- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd4e8004d40 msgr2=0x7fd4fc14a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5008eb640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd4e8004d40 0x7fd4fc14a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5008eb640 1 -- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0c3af0 msgr2=0x7fd4fc0cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5008eb640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0c3af0 0x7fd4fc0cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5008eb640 1 -- 172.21.15.117:0/2206779656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc15ad80 con 0x7fd4fc149960 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd5008eb640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 0x7fd4fc0c35b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0066b0 tx=0x7fd4ec0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:30.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec015070 con 0x7fd4fc149960 2024-09-17T05:27:30.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4ec00f960 con 0x7fd4fc149960 2024-09-17T05:27:30.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec00fdb0 con 0x7fd4fc149960 2024-09-17T05:27:30.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4fc15b060 con 0x7fd4fc149960 2024-09-17T05:27:30.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.806+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4fc14b480 con 0x7fd4fc149960 2024-09-17T05:27:30.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.807+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4fc0c7940 con 0x7fd4fc149960 2024-09-17T05:27:30.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.808+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd4ec011430 con 0x7fd4fc149960 2024-09-17T05:27:30.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.808+0000 7fd4f1ffb640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd4e0078b90 0x7fd4e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:30.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.808+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7fd4ec09bd50 con 0x7fd4fc149960 2024-09-17T05:27:30.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.809+0000 7fd5010ec640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd4e0078b90 0x7fd4e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:30.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.809+0000 7fd5010ec640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd4e0078b90 0x7fd4e007b0a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8003700 tx=0x7fd4f800b260 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:30.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:30.813+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4ec09f030 con 0x7fd4fc149960 2024-09-17T05:27:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.027+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fd4fc0c4750 con 0x7fd4fc149960 2024-09-17T05:27:31.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.688+0000 7fd4f1ffb640 1 -- 172.21.15.117:0/2206779656 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v5) ==== 137+0+0 (secure 0 0 0) 0x7fd4ec063f10 con 0x7fd4fc149960 2024-09-17T05:27:31.688 INFO:teuthology.orchestra.run.smithi117.stderr:added data pool 4 to fsmap 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd4e0078b90 msgr2=0x7fd4e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd4e0078b90 0x7fd4e007b0a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8003700 tx=0x7fd4f800b260 comp rx=0 tx=0).stop 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 msgr2=0x7fd4fc0c35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 0x7fd4fc0c35b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0066b0 tx=0x7fd4ec0039c0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 shutdown_connections 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd4e0078b90 0x7fd4e007b0a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0c3af0 0x7fd4fc0cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4fc149960 0x7fd4fc0c35b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 --2- 172.21.15.117:0/2206779656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd4e8004d40 0x7fd4fc14a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 >> 172.21.15.117:0/2206779656 conn(0x7fd4fc06e590 msgr2=0x7fd4fc1578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:31.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 shutdown_connections 2024-09-17T05:27:31.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.690+0000 7fd503350640 1 -- 172.21.15.117:0/2206779656 wait complete. 2024-09-17T05:27:31.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:31.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 -- 172.21.15.117:0/1276822087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c67a0 msgr2=0x7f2a94152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1276822087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c67a0 0x7f2a94152210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005fd0 tx=0x7f2a8402a820 comp rx=0 tx=0).stop 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 -- 172.21.15.117:0/1276822087 shutdown_connections 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1276822087 >> v2:172.21.15.119:3300/0 conn(0x7f2a940c5660 0x7f2a94071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1276822087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c67a0 0x7f2a94152210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 -- 172.21.15.117:0/1276822087 >> 172.21.15.117:0/1276822087 conn(0x7f2a9406e590 msgr2=0x7f2a940cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 -- 172.21.15.117:0/1276822087 shutdown_connections 2024-09-17T05:27:31.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.818+0000 7f2a9c139640 1 -- 172.21.15.117:0/1276822087 wait complete. 2024-09-17T05:27:31.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.820+0000 7f2a9c139640 1 Processor -- start 2024-09-17T05:27:31.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.820+0000 7f2a9c139640 1 -- start start 2024-09-17T05:27:31.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9c139640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a94071850 0x7f2a940cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:31.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9c139640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c5660 0x7f2a941dee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9c139640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 0x7f2a941e2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9c139640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940c7ff0 con 0x7f2a940c67a0 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9c139640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940ce0d0 con 0x7f2a94071850 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9c139640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2a940c83a0 con 0x7f2a940c5660 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 0x7f2a941e2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 0x7f2a941e2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40990/0 (socket says 172.21.15.117:40990) 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 -- 172.21.15.117:0/1932172246 learned_addr learned my addr 172.21.15.117:0/1932172246 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a996d4640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c5660 0x7f2a941dee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a99ed5640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a94071850 0x7f2a940cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 -- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c5660 msgr2=0x7f2a941dee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c5660 0x7f2a941dee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 -- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a94071850 msgr2=0x7f2a940cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a94071850 0x7f2a940cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.821+0000 7f2a9a6d6640 1 -- 172.21.15.117:0/1932172246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a940cbfc0 con 0x7f2a940c67a0 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.822+0000 7f2a9a6d6640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 0x7f2a941e2740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00e900 tx=0x7f2a8c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.822+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00cd80 con 0x7f2a940c67a0 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.822+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a8c004020 con 0x7f2a940c67a0 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.822+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c004450 con 0x7f2a940c67a0 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.822+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a94146570 con 0x7f2a940c67a0 2024-09-17T05:27:31.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.822+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a940cc1d0 con 0x7f2a940c67a0 2024-09-17T05:27:31.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.823+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a941467e0 con 0x7f2a940c67a0 2024-09-17T05:27:31.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.823+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a8c0035f0 con 0x7f2a940c67a0 2024-09-17T05:27:31.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.824+0000 7f2a927fc640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a80078bc0 0x7f2a8007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:31.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.824+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a8c09b7f0 con 0x7f2a940c67a0 2024-09-17T05:27:31.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.824+0000 7f2a99ed5640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a80078bc0 0x7f2a8007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:31.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.824+0000 7f2a99ed5640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a80078bc0 0x7f2a8007b0d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a84034000 tx=0x7f2a84004ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:31.828+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a8c0633c0 con 0x7f2a940c67a0 2024-09-17T05:27:32.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.037+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a940c7940 con 0x7f2a940c67a0 2024-09-17T05:27:32.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.038+0000 7f2a927fc640 1 -- 172.21.15.117:0/1932172246 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+6879 (secure 0 0 0) 0x7f2a8c067df0 con 0x7f2a940c67a0 2024-09-17T05:27:32.039 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:32.039 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"btime":"2024-09-17T05:27:31:691208+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4434,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:31.681445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:27:32.041 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 6 2024-09-17T05:27:32.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a80078bc0 msgr2=0x7f2a8007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a80078bc0 0x7f2a8007b0d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a84034000 tx=0x7f2a84004ab0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 msgr2=0x7f2a941e2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 0x7f2a941e2740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00e900 tx=0x7f2a8c00edc0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 shutdown_connections 2024-09-17T05:27:32.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a80078bc0 0x7f2a8007b0d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a940c67a0 0x7f2a941e2740 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a940c5660 0x7f2a941dee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 --2- 172.21.15.117:0/1932172246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a94071850 0x7f2a940cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 >> 172.21.15.117:0/1932172246 conn(0x7f2a9406e590 msgr2=0x7f2a9414e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:32.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 shutdown_connections 2024-09-17T05:27:32.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.040+0000 7f2a9c139640 1 -- 172.21.15.117:0/1932172246 wait complete. 2024-09-17T05:27:32.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 -- 172.21.15.117:0/653790006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c06dbe0 msgr2=0x7f0a3c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 --2- 172.21.15.117:0/653790006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c06dbe0 0x7f0a3c06dff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a34005fd0 tx=0x7f0a3402a8d0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 -- 172.21.15.117:0/653790006 shutdown_connections 2024-09-17T05:27:32.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 --2- 172.21.15.117:0/653790006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c06dbe0 0x7f0a3c06dff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 --2- 172.21.15.117:0/653790006 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c0b52b0 0x7f0a3c0b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 -- 172.21.15.117:0/653790006 >> 172.21.15.117:0/653790006 conn(0x7f0a3c074d80 msgr2=0x7f0a3c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:32.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 -- 172.21.15.117:0/653790006 shutdown_connections 2024-09-17T05:27:32.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.165+0000 7f0a416f2640 1 -- 172.21.15.117:0/653790006 wait complete. 2024-09-17T05:27:32.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.167+0000 7f0a416f2640 1 Processor -- start 2024-09-17T05:27:32.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.167+0000 7f0a416f2640 1 -- start start 2024-09-17T05:27:32.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.168+0000 7f0a416f2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c06dbe0 0x7f0a3c15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.168+0000 7f0a416f2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c0b52b0 0x7f0a3c0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.168+0000 7f0a416f2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 0x7f0a3c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.168+0000 7f0a416f2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c1558b0 con 0x7f0a3c15bd70 2024-09-17T05:27:32.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.168+0000 7f0a416f2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c0cd0d0 con 0x7f0a3c0b52b0 2024-09-17T05:27:32.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.168+0000 7f0a416f2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3c0cb290 con 0x7f0a3c06dbe0 2024-09-17T05:27:32.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 0x7f0a3c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 0x7f0a3c15ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41000/0 (socket says 172.21.15.117:41000) 2024-09-17T05:27:32.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 -- 172.21.15.117:0/3908128536 learned_addr learned my addr 172.21.15.117:0/3908128536 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:32.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a327fc640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c0b52b0 0x7f0a3c0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3affd640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c06dbe0 0x7f0a3c15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 -- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c06dbe0 msgr2=0x7f0a3c15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c06dbe0 0x7f0a3c15cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 -- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c0b52b0 msgr2=0x7f0a3c0c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c0b52b0 0x7f0a3c0c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 -- 172.21.15.117:0/3908128536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c1574c0 con 0x7f0a3c15bd70 2024-09-17T05:27:32.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3affd640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c06dbe0 0x7f0a3c15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:32.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a3b7fe640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 0x7f0a3c15ddc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009cd0 tx=0x7f0a34032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:32.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a34037040 con 0x7f0a3c15bd70 2024-09-17T05:27:32.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a34003c60 con 0x7f0a3c15bd70 2024-09-17T05:27:32.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3c15d100 con 0x7f0a3c15bd70 2024-09-17T05:27:32.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3403c990 con 0x7f0a3c15bd70 2024-09-17T05:27:32.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.169+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3c1567a0 con 0x7f0a3c15bd70 2024-09-17T05:27:32.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.170+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a3c0ccc90 con 0x7f0a3c15bd70 2024-09-17T05:27:32.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.171+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0a3403a070 con 0x7f0a3c15bd70 2024-09-17T05:27:32.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.172+0000 7f0a33fff640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0a1c078bc0 0x7f0a1c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.172+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0a340bbda0 con 0x7f0a3c15bd70 2024-09-17T05:27:32.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.172+0000 7f0a3affd640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0a1c078bc0 0x7f0a1c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.172+0000 7f0a3affd640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0a1c078bc0 0x7f0a1c07b0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c007960 tx=0x7f0a2c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:32.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.176+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0a34041020 con 0x7f0a3c15bd70 2024-09-17T05:27:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.352+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a3c0cc9b0 con 0x7f0a3c15bd70 2024-09-17T05:27:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.353+0000 7f0a33fff640 1 -- 172.21.15.117:0/3908128536 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f0a340838e0 con 0x7f0a3c15bd70 2024-09-17T05:27:32.354 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:32.355 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0a1c078bc0 msgr2=0x7f0a1c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0a1c078bc0 0x7f0a1c07b0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c007960 tx=0x7f0a2c009040 comp rx=0 tx=0).stop 2024-09-17T05:27:32.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 msgr2=0x7f0a3c15ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 0x7f0a3c15ddc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009cd0 tx=0x7f0a34032f30 comp rx=0 tx=0).stop 2024-09-17T05:27:32.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 shutdown_connections 2024-09-17T05:27:32.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0a1c078bc0 0x7f0a1c07b0d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a3c15bd70 0x7f0a3c15ddc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a3c0b52b0 0x7f0a3c0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.355+0000 7f0a416f2640 1 --2- 172.21.15.117:0/3908128536 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0a3c06dbe0 0x7f0a3c15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.356+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 >> 172.21.15.117:0/3908128536 conn(0x7f0a3c074d80 msgr2=0x7f0a3c06e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.356+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 shutdown_connections 2024-09-17T05:27:32.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.356+0000 7f0a416f2640 1 -- 172.21.15.117:0/3908128536 wait complete. 2024-09-17T05:27:32.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:32.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f931b369640 1 -- 172.21.15.117:0/2235448439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9300003a00 msgr2=0x7f9300003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f931b369640 1 --2- 172.21.15.117:0/2235448439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9300003a00 0x7f9300003e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9310009b50 tx=0x7f9300005830 comp rx=0 tx=0).stop 2024-09-17T05:27:32.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f9319906640 1 -- 172.21.15.117:0/2235448439 reap_dead start 2024-09-17T05:27:32.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f931b369640 1 -- 172.21.15.117:0/2235448439 shutdown_connections 2024-09-17T05:27:32.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f931b369640 1 -- 172.21.15.117:0/2235448439 >> 172.21.15.117:0/2235448439 conn(0x7f9314075e00 msgr2=0x7f931406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:32.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f931b369640 1 -- 172.21.15.117:0/2235448439 shutdown_connections 2024-09-17T05:27:32.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.481+0000 7f931b369640 1 -- 172.21.15.117:0/2235448439 wait complete. 2024-09-17T05:27:32.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.483+0000 7f931b369640 1 Processor -- start 2024-09-17T05:27:32.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.483+0000 7f931b369640 1 -- start start 2024-09-17T05:27:32.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f931b369640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300003a00 0x7f9314152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f931b369640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9314153de0 0x7f93140b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f931b369640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 0x7f931414f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f931b369640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f93140c5da0 con 0x7f931414f270 2024-09-17T05:27:32.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f931b369640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f931415f6b0 con 0x7f9300003a00 2024-09-17T05:27:32.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f931b369640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f93141558b0 con 0x7f9314153de0 2024-09-17T05:27:32.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f9319906640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 0x7f931414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f9319906640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 0x7f931414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41020/0 (socket says 172.21.15.117:41020) 2024-09-17T05:27:32.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f9319906640 1 -- 172.21.15.117:0/338962276 learned_addr learned my addr 172.21.15.117:0/338962276 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:32.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f9319105640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300003a00 0x7f9314152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.484+0000 7f9318904640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9314153de0 0x7f93140b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9319906640 1 -- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9314153de0 msgr2=0x7f93140b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9319906640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9314153de0 0x7f93140b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9319906640 1 -- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300003a00 msgr2=0x7f9314152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9319906640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300003a00 0x7f9314152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9319906640 1 -- 172.21.15.117:0/338962276 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931414fbc0 con 0x7f931414f270 2024-09-17T05:27:32.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9319906640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 0x7f931414f680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f930c010ab0 tx=0x7f930c010eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:32.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c00ee20 con 0x7f931414f270 2024-09-17T05:27:32.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f930c015070 con 0x7f931414f270 2024-09-17T05:27:32.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c003d00 con 0x7f931414f270 2024-09-17T05:27:32.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9314154eb0 con 0x7f931414f270 2024-09-17T05:27:32.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.485+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9314154bd0 con 0x7f931414f270 2024-09-17T05:27:32.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.487+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93140c7a30 con 0x7f931414f270 2024-09-17T05:27:32.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.488+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f930c017020 con 0x7f931414f270 2024-09-17T05:27:32.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.489+0000 7f9309ffb640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f92f8078bc0 0x7f92f807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.489+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f930c09aed0 con 0x7f931414f270 2024-09-17T05:27:32.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.490+0000 7f9319105640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f92f8078bc0 0x7f92f807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.490+0000 7f9319105640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f92f8078bc0 0x7f92f807b0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9310009b50 tx=0x7f931002d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:32.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.494+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f930c020070 con 0x7f931414f270 2024-09-17T05:27:32.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.671+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9314158f10 con 0x7f931414f270 2024-09-17T05:27:32.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.672+0000 7f9309ffb640 1 -- 172.21.15.117:0/338962276 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f931414fbc0 con 0x7f931414f270 2024-09-17T05:27:32.672 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:32.672 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:32.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f92f8078bc0 msgr2=0x7f92f807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f92f8078bc0 0x7f92f807b0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9310009b50 tx=0x7f931002d040 comp rx=0 tx=0).stop 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 msgr2=0x7f931414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 0x7f931414f680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f930c010ab0 tx=0x7f930c010eb0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 shutdown_connections 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f92f8078bc0 0x7f92f807b0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f931414f270 0x7f931414f680 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9314153de0 0x7f93140b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 --2- 172.21.15.117:0/338962276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300003a00 0x7f9314152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 >> 172.21.15.117:0/338962276 conn(0x7f9314075e00 msgr2=0x7f93140d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 shutdown_connections 2024-09-17T05:27:32.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.674+0000 7f931b369640 1 -- 172.21.15.117:0/338962276 wait complete. 2024-09-17T05:27:32.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:32.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 -- 172.21.15.117:0/765696106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 msgr2=0x7efe480d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 --2- 172.21.15.117:0/765696106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe480d0e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe480763a0 tx=0x7efe3c02a5f0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 -- 172.21.15.117:0/765696106 shutdown_connections 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 --2- 172.21.15.117:0/765696106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe480d0e30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 --2- 172.21.15.117:0/765696106 >> v2:172.21.15.119:3300/0 conn(0x7efe480c62c0 0x7efe480c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 --2- 172.21.15.117:0/765696106 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efe480cb390 0x7efe480d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 -- 172.21.15.117:0/765696106 >> 172.21.15.117:0/765696106 conn(0x7efe48075e00 msgr2=0x7efe4806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 -- 172.21.15.117:0/765696106 shutdown_connections 2024-09-17T05:27:32.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.814+0000 7efe4f24d640 1 -- 172.21.15.117:0/765696106 wait complete. 2024-09-17T05:27:32.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.815+0000 7efe4f24d640 1 Processor -- start 2024-09-17T05:27:32.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.816+0000 7efe4f24d640 1 -- start start 2024-09-17T05:27:32.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4f24d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe480c62c0 0x7efe481f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4f24d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efe480cb390 0x7efe48151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4f24d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe48157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4f24d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efe480c5da0 con 0x7efe4815b2f0 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4f24d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efe4815f6b0 con 0x7efe480c62c0 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4f24d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7efe480c6c10 con 0x7efe480cb390 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe48157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe48157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41032/0 (socket says 172.21.15.117:41032) 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 -- 172.21.15.117:0/2201903071 learned_addr learned my addr 172.21.15.117:0/2201903071 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4cfe9640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe480c62c0 0x7efe481f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe47fff640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efe480cb390 0x7efe48151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 -- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efe480cb390 msgr2=0x7efe48151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efe480cb390 0x7efe48151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 -- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe480c62c0 msgr2=0x7efe481f16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe480c62c0 0x7efe481f16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 -- 172.21.15.117:0/2201903071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe481f1bf0 con 0x7efe4815b2f0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4cfe9640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe480c62c0 0x7efe481f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.817+0000 7efe4d7ea640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe48157980 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe4000cae0 tx=0x7efe40007220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.818+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40007da0 con 0x7efe4815b2f0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.818+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe40004c30 con 0x7efe4815b2f0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.818+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40005280 con 0x7efe4815b2f0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.818+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe48158ab0 con 0x7efe4815b2f0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.818+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe48155cc0 con 0x7efe4815b2f0 2024-09-17T05:27:32.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.819+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe480c8ad0 con 0x7efe4815b2f0 2024-09-17T05:27:32.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.819+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efe40005420 con 0x7efe4815b2f0 2024-09-17T05:27:32.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.820+0000 7efe457fa640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7efe34078bc0 0x7efe3407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:32.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.820+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efe4009ba70 con 0x7efe4815b2f0 2024-09-17T05:27:32.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.820+0000 7efe4cfe9640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7efe34078bc0 0x7efe3407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:32.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.821+0000 7efe4cfe9640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7efe34078bc0 0x7efe3407b0d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe38009890 tx=0x7efe38008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:32.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:32.824+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efe40014070 con 0x7efe4815b2f0 2024-09-17T05:27:33.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.039+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe480cb810 con 0x7efe4815b2f0 2024-09-17T05:27:33.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.040+0000 7efe457fa640 1 -- 172.21.15.117:0/2201903071 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+6879 (secure 0 0 0) 0x7efe40063c00 con 0x7efe4815b2f0 2024-09-17T05:27:33.041 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:33.041 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"btime":"2024-09-17T05:27:31:691208+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4434,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:31.681445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:27:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 6 2024-09-17T05:27:33.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7efe34078bc0 msgr2=0x7efe3407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7efe34078bc0 0x7efe3407b0d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efe38009890 tx=0x7efe38008040 comp rx=0 tx=0).stop 2024-09-17T05:27:33.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 msgr2=0x7efe48157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe48157980 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe4000cae0 tx=0x7efe40007220 comp rx=0 tx=0).stop 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 shutdown_connections 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7efe34078bc0 0x7efe3407b0d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe4815b2f0 0x7efe48157980 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efe480cb390 0x7efe48151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 --2- 172.21.15.117:0/2201903071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe480c62c0 0x7efe481f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 >> 172.21.15.117:0/2201903071 conn(0x7efe48075e00 msgr2=0x7efe4815c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:33.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 shutdown_connections 2024-09-17T05:27:33.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.042+0000 7efe4f24d640 1 -- 172.21.15.117:0/2201903071 wait complete. 2024-09-17T05:27:33.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:33.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 -- 172.21.15.117:0/1143805451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd8153210 msgr2=0x7fbdd8159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 --2- 172.21.15.117:0/1143805451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd8153210 0x7fbdd8159a90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4005fd0 tx=0x7fbdd402a5d0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 -- 172.21.15.117:0/1143805451 shutdown_connections 2024-09-17T05:27:33.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 --2- 172.21.15.117:0/1143805451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd8153210 0x7fbdd8159a90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 -- 172.21.15.117:0/1143805451 >> 172.21.15.117:0/1143805451 conn(0x7fbdd8072cc0 msgr2=0x7fbdd80730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:33.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 -- 172.21.15.117:0/1143805451 shutdown_connections 2024-09-17T05:27:33.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.165+0000 7fbde05ad640 1 -- 172.21.15.117:0/1143805451 wait complete. 2024-09-17T05:27:33.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.166+0000 7fbde05ad640 1 Processor -- start 2024-09-17T05:27:33.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.166+0000 7fbde05ad640 1 -- start start 2024-09-17T05:27:33.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdd80cb080 0x7fbdd8144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 0x7fbdd8143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdd8153210 0x7fbdd806deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd815d0d0 con 0x7fbdd80cb490 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8159fd0 con 0x7fbdd80cb080 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd80c3ce0 con 0x7fbdd8153210 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 0x7fbdd8143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 0x7fbdd8143f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41048/0 (socket says 172.21.15.117:41048) 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 -- 172.21.15.117:0/3860441928 learned_addr learned my addr 172.21.15.117:0/3860441928 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdde349640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdd80cb080 0x7fbdd8144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbddeb4a640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdd8153210 0x7fbdd806deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 -- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdd8153210 msgr2=0x7fbdd806deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdd8153210 0x7fbdd806deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 -- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdd80cb080 msgr2=0x7fbdd8144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdd80cb080 0x7fbdd8144d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 -- 172.21.15.117:0/3860441928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd81447d0 con 0x7fbdd80cb490 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdde349640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdd80cb080 0x7fbdd8144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdddb48640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 0x7fbdd8143f40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c390 tx=0x7fbdc800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8017070 con 0x7fbdd80cb490 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdc80053c0 con 0x7fbdd80cb490 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd81452a0 con 0x7fbdd80cb490 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8003850 con 0x7fbdd80cb490 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.168+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd81f2400 con 0x7fbdd80cb490 2024-09-17T05:27:33.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.169+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd8159a90 con 0x7fbdd80cb490 2024-09-17T05:27:33.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.170+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbdc801a070 con 0x7fbdd80cb490 2024-09-17T05:27:33.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.171+0000 7fbdceffd640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:33.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.171+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbdc809a3d0 con 0x7fbdd80cb490 2024-09-17T05:27:33.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.171+0000 7fbdde349640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:33.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.172+0000 7fbdde349640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4033000 tx=0x7fbdd4004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:33.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.176+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbdd81447d0 con 0x7fbdd80cb490 2024-09-17T05:27:33.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.348+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbdd80ca1a0 con 0x7fbdd80cb490 2024-09-17T05:27:33.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.349+0000 7fbdceffd640 1 -- 172.21.15.117:0/3860441928 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7fbdc8062560 con 0x7fbdd80cb490 2024-09-17T05:27:33.350 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:33.350 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbdbc078b90 msgr2=0x7fbdbc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4033000 tx=0x7fbdd4004730 comp rx=0 tx=0).stop 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 msgr2=0x7fbdd8143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 0x7fbdd8143f40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c390 tx=0x7fbdc800c850 comp rx=0 tx=0).stop 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 shutdown_connections 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdd8153210 0x7fbdd806deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd80cb490 0x7fbdd8143f40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 --2- 172.21.15.117:0/3860441928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdd80cb080 0x7fbdd8144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:33.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.351+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 >> 172.21.15.117:0/3860441928 conn(0x7fbdd8072cc0 msgr2=0x7fbdd80c95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:33.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.352+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 shutdown_connections 2024-09-17T05:27:33.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:33.352+0000 7fbde05ad640 1 -- 172.21.15.117:0/3860441928 wait complete. 2024-09-17T05:27:33.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T05:27:26.003433+0000', 'modified': '2024-09-17T05:27:31.681445+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4440}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4440': {'gid': 4440, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6839/2500745218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2500745218}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2500745218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4440, 'qdb_cluster': [4440]} 2024-09-17T05:27:33.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T05:27:33.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-09-17T05:27:34.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb776fed640 1 -- 172.21.15.117:0/1553319532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 msgr2=0x7fb760003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb776fed640 1 --2- 172.21.15.117:0/1553319532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb760003e70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c290 tx=0x7fb760005830 comp rx=0 tx=0).stop 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb77558a640 1 -- 172.21.15.117:0/1553319532 reap_dead start 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb776fed640 1 -- 172.21.15.117:0/1553319532 shutdown_connections 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb776fed640 1 -- 172.21.15.117:0/1553319532 >> 172.21.15.117:0/1553319532 conn(0x7fb7700744d0 msgr2=0x7fb7700748e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb776fed640 1 -- 172.21.15.117:0/1553319532 shutdown_connections 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.082+0000 7fb776fed640 1 -- 172.21.15.117:0/1553319532 wait complete. 2024-09-17T05:27:34.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.084+0000 7fb776fed640 1 Processor -- start 2024-09-17T05:27:34.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.084+0000 7fb776fed640 1 -- start start 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb776fed640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb77014d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb776fed640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb77014ba70 0x7fb7701f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb776fed640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb77014a200 0x7fb77014a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb776fed640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb770151fa0 con 0x7fb760003a00 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb776fed640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb77015f2b0 con 0x7fb77014a200 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb776fed640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb770152120 con 0x7fb77014ba70 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb774d89640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb77014d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb774d89640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb77014d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50498/0 (socket says 172.21.15.117:50498) 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.085+0000 7fb774d89640 1 -- 172.21.15.117:0/4064424202 learned_addr learned my addr 172.21.15.117:0/4064424202 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb76ffff640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb77014ba70 0x7fb7701f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb77558a640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb77014a200 0x7fb77014a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb774d89640 1 -- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb77014ba70 msgr2=0x7fb7701f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb774d89640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb77014ba70 0x7fb7701f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb774d89640 1 -- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb77014a200 msgr2=0x7fb77014a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb774d89640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb77014a200 0x7fb77014a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb774d89640 1 -- 172.21.15.117:0/4064424202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77014d700 con 0x7fb760003a00 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb774d89640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb77014d1c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c380 tx=0x7fb764001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb76d7fa640 1 -- 172.21.15.117:0/4064424202 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764031040 con 0x7fb760003a00 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb76d7fa640 1 -- 172.21.15.117:0/4064424202 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7640035f0 con 0x7fb760003a00 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7700c63a0 con 0x7fb760003a00 2024-09-17T05:27:34.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb76d7fa640 1 -- 172.21.15.117:0/4064424202 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764003a00 con 0x7fb760003a00 2024-09-17T05:27:34.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.086+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7701f5980 con 0x7fb760003a00 2024-09-17T05:27:34.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.088+0000 7fb76d7fa640 1 -- 172.21.15.117:0/4064424202 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb764034070 con 0x7fb760003a00 2024-09-17T05:27:34.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.089+0000 7fb76d7fa640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078b90 0x7fb75407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.089+0000 7fb76d7fa640 1 -- 172.21.15.117:0/4064424202 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb7640bb0e0 con 0x7fb760003a00 2024-09-17T05:27:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.089+0000 7fb76ffff640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078b90 0x7fb75407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.090+0000 7fb77e037740 1 Processor -- start 2024-09-17T05:27:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.090+0000 7fb76ffff640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078b90 0x7fb75407b0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb7700b4c80 tx=0x7fb75c005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:34.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.090+0000 7fb77e037740 1 -- start start 2024-09-17T05:27:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca321940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77e037740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca326b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77e037740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643ca299820 con 0x5643c9f5e590 2024-09-17T05:27:34.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5643ca288c50 con 0x5643c9f563b0 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x5643ca28ca60 con 0x5643c9f7f270 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca326b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca326b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50504/0 (socket says 172.21.15.117:50504) 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 -- 172.21.15.117:0/84595668 learned_addr learned my addr 172.21.15.117:0/84595668 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb774d89640 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb76ffff640 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca321940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 -- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 msgr2=0x5643ca36b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 -- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 msgr2=0x5643ca321940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca321940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 -- 172.21.15.117:0/84595668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643c9fea850 con 0x5643c9f5e590 2024-09-17T05:27:34.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb77558a640 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca326b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb768005fd0 tx=0x7fb768012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885c442ca190d1c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.091+0000 7fb75a7fc640 1 -- 172.21.15.117:0/84595668 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7680120d0 con 0x5643c9f5e590 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb75a7fc640 1 -- 172.21.15.117:0/84595668 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb768012270 con 0x5643c9f5e590 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb75a7fc640 1 -- 172.21.15.117:0/84595668 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768003c90 con 0x5643c9f5e590 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 -- 172.21.15.117:0/84595668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 msgr2=0x5643ca326b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca326b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb768005fd0 tx=0x7fb768012820 comp rx=0 tx=0).stop 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 -- 172.21.15.117:0/84595668 shutdown_connections 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca326b50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca321940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 --2- 172.21.15.117:0/84595668 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 -- 172.21.15.117:0/84595668 >> 172.21.15.117:0/84595668 conn(0x5643c9f733c0 msgr2=0x5643ca36a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 -- 172.21.15.117:0/84595668 shutdown_connections 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.092+0000 7fb77e037740 1 -- 172.21.15.117:0/84595668 wait complete. 2024-09-17T05:27:34.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.093+0000 7fb77e037740 1 -- start start 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.093+0000 7fb77e037740 1 Processor -- start 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca35be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca359540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca35f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643c9ff9620 con 0x5643c9f5e590 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5643ca22e5f0 con 0x5643c9f563b0 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x5643ca359f50 con 0x5643c9f7f270 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 10 client.0 _ll_register_callbacks cb 0x5643ca2320a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca359540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca359540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50506/0 (socket says 172.21.15.117:50506) 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 -- 172.21.15.117:0/3048118630 learned_addr learned my addr 172.21.15.117:0/3048118630 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb774d89640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca35be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77558a640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca35f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 -- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 msgr2=0x5643ca35f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca35f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 -- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 msgr2=0x5643ca35be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca35be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 -- 172.21.15.117:0/3048118630 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643ca0616b0 con 0x5643c9f5e590 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb76ffff640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca359540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb75c01da20 tx=0x7fb75c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272b42387564b94f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb74f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3048118630 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c02e060 con 0x5643c9f5e590 2024-09-17T05:27:34.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3048118630 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb75c00f5d0 con 0x5643c9f5e590 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.095+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb768012270 con 0x5643c9f5e590 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3048118630 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fb75c034050 con 0x5643c9f5e590 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 10 client.4542 fetch_fsmap learned FSMap version 6 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x5643ca321390 con 0x5643c9f5e590 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3048118630 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb768012270 con 0x5643c9f5e590 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3048118630 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 6) ==== 4720+0+0 (secure 0 0 0) 0x7fb75c037020 con 0x5643c9f5e590 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 10 client.4542 fetch_fsmap finished waiting for FSMap version 6 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 10 client.4542 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 20 client.4542 populate_metadata read hostname 'smithi117' 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643c9f3de40 0x5643ca385700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 4 client.4542 mds_command: new command op to 4440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 --> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643ca29b770 con 0x5643c9f3de40 2024-09-17T05:27:34.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.096+0000 7fb774d89640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643c9f3de40 0x5643ca385700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:34.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.097+0000 7fb74f7fe640 10 client.4542 ms_handle_connect on v2:172.21.15.119:6838/2500745218 2024-09-17T05:27:34.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:34.097+0000 7fb774d89640 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643c9f3de40 0x5643ca385700 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479a3a769e80d9cc server_cookie=80223fc2908944b3 in_seq=0 out_seq=0 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.690+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3048118630 <== mds.0 v2:172.21.15.119:6838/2500745218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5643ca29b770 con 0x5643c9f3de40 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.690+0000 7fb74f7fe640 10 client.4542 handle_command_reply: tid=0 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.690+0000 7fb77e037740 1 client.4542 shutdown 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 msgr2=0x5643ca359540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca359540 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb75c01da20 tx=0x7fb75c0046c0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 shutdown_connections 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643c9f3de40 0x5643ca385700 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca35f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca359540 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 --2- 172.21.15.117:0/3048118630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca35be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 >> 172.21.15.117:0/3048118630 conn(0x5643c9f733c0 msgr2=0x5643ca328180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:36.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb74f7fe640 0 client.4542 ms_handle_reset on v2:172.21.15.119:6838/2500745218 2024-09-17T05:27:36.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 shutdown_connections 2024-09-17T05:27:36.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 1 -- 172.21.15.117:0/3048118630 wait complete. 2024-09-17T05:27:36.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.691+0000 7fb77e037740 20 client.4542 trim_cache size 0 max 16384 2024-09-17T05:27:36.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.711+0000 7fb77e037740 1 Processor -- start 2024-09-17T05:27:36.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.711+0000 7fb77e037740 1 -- start start 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca360e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb77e037740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca361360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb77e037740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7701febb0 con 0x5643c9f5e590 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5643ca359c90 con 0x5643c9f563b0 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x5643ca35b870 con 0x5643c9f7f270 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb76ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca361360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb76ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca361360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50510/0 (socket says 172.21.15.117:50510) 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.712+0000 7fb76ffff640 1 -- 172.21.15.117:0/950410578 learned_addr learned my addr 172.21.15.117:0/950410578 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb77558a640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb774d89640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca360e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb76ffff640 1 -- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 msgr2=0x5643ca36a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb76ffff640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb76ffff640 1 -- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 msgr2=0x5643ca360e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb76ffff640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca360e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb76ffff640 1 -- 172.21.15.117:0/950410578 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643ca368000 con 0x5643c9f5e590 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb77558a640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb774d89640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca360e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:36.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb76ffff640 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca361360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb75c019040 tx=0x7fb75c012d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501d436021f0f286 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:36.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb759ffb640 1 -- 172.21.15.117:0/950410578 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00f5d0 con 0x5643c9f5e590 2024-09-17T05:27:36.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb759ffb640 1 -- 172.21.15.117:0/950410578 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb75c00fbc0 con 0x5643c9f5e590 2024-09-17T05:27:36.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.713+0000 7fb759ffb640 1 -- 172.21.15.117:0/950410578 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c005020 con 0x5643c9f5e590 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 -- 172.21.15.117:0/950410578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 msgr2=0x5643ca361360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca361360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb75c019040 tx=0x7fb75c012d80 comp rx=0 tx=0).stop 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 -- 172.21.15.117:0/950410578 shutdown_connections 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f7f270 0x5643ca36a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f5e590 0x5643ca361360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 --2- 172.21.15.117:0/950410578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f563b0 0x5643ca360e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 -- 172.21.15.117:0/950410578 >> 172.21.15.117:0/950410578 conn(0x5643c9f733c0 msgr2=0x5643ca322bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 -- 172.21.15.117:0/950410578 shutdown_connections 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.714+0000 7fb77e037740 1 -- 172.21.15.117:0/950410578 wait complete. 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.715+0000 7fb77e037740 1 -- start start 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.716+0000 7fb77e037740 1 Processor -- start 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f563b0 0x5643ca373560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f5e590 0x5643ca3e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 0x5643ca3e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643ca367e80 con 0x5643c9f7f270 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5643ca3677f0 con 0x5643c9f5e590 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x5643ca37a320 con 0x5643c9f563b0 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77e037740 10 client.0 _ll_register_callbacks cb 0x7fb75c012c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 0x5643ca3e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 0x5643ca3e49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50514/0 (socket says 172.21.15.117:50514) 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 -- 172.21.15.117:0/3068288662 learned_addr learned my addr 172.21.15.117:0/3068288662 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb774d89640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f563b0 0x5643ca373560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb76ffff640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f5e590 0x5643ca3e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 -- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f563b0 msgr2=0x5643ca373560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f563b0 0x5643ca373560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 -- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f5e590 msgr2=0x5643ca3e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f5e590 0x5643ca3e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.717+0000 7fb77558a640 1 -- 172.21.15.117:0/3068288662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643ca358950 con 0x5643c9f7f270 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77558a640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 0x5643ca3e49c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb768012d20 tx=0x7fb768010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16dbb7e30a9ca9b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb74f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3068288662 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76802a060 con 0x5643c9f7f270 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3068288662 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb768010a10 con 0x5643c9f7f270 2024-09-17T05:27:36.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb75c00fbc0 con 0x5643c9f7f270 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3068288662 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fb768030050 con 0x5643c9f7f270 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 10 client.4548 fetch_fsmap learned FSMap version 6 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x5643ca325af0 con 0x5643c9f7f270 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3068288662 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb75c00fbc0 con 0x5643c9f7f270 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3068288662 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 6) ==== 4720+0+0 (secure 0 0 0) 0x7fb768033020 con 0x5643c9f7f270 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 10 client.4548 fetch_fsmap finished waiting for FSMap version 6 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 10 client.4548 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 20 client.4548 populate_metadata read hostname 'smithi117' 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643ca36fd70 0x5643ca365ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 4 client.4548 mds_command: new command op to 4440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.718+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 --> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] -- command(tid 0: {"prefix": "status"}) -- 0x5643ca297220 con 0x5643ca36fd70 2024-09-17T05:27:36.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.719+0000 7fb774d89640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643ca36fd70 0x5643ca365ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.719+0000 7fb74f7fe640 10 client.4548 ms_handle_connect on v2:172.21.15.119:6838/2500745218 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.719+0000 7fb774d89640 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643ca36fd70 0x5643ca365ed0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f07a7ed4b2532fbb server_cookie=973ca4a4bfca6d5f in_seq=0 out_seq=0 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.719+0000 7fb74f7fe640 1 -- 172.21.15.117:0/3068288662 <== mds.0 v2:172.21.15.119:6838/2500745218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5643ca297220 con 0x5643ca36fd70 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.719+0000 7fb74f7fe640 10 client.4548 handle_command_reply: tid=0 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.719+0000 7fb77e037740 1 client.4548 shutdown 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 msgr2=0x5643ca3e49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 0x5643ca3e49c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb768012d20 tx=0x7fb768010660 comp rx=0 tx=0).stop 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 shutdown_connections 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x5643ca36fd70 0x5643ca365ed0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643c9f7f270 0x5643ca3e49c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643c9f5e590 0x5643ca3e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 --2- 172.21.15.117:0/3068288662 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x5643c9f563b0 0x5643ca373560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb74f7fe640 0 client.4548 ms_handle_reset on v2:172.21.15.119:6838/2500745218 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 >> 172.21.15.117:0/3068288662 conn(0x5643c9f733c0 msgr2=0x5643ca35a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 shutdown_connections 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 1 -- 172.21.15.117:0/3068288662 wait complete. 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.720+0000 7fb77e037740 20 client.4548 trim_cache size 0 max 16384 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "8527add3-c46f-4f7f-86e8-5cd706a0043e", 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 0, 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 4440, 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-09-17T05:27:36.726 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 10.083425332999999, 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 5, 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 25, 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 25, 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 16.084007022000002 2024-09-17T05:27:36.727 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:27:36.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078b90 msgr2=0x7fb75407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078b90 0x7fb75407b0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb7700b4c80 tx=0x7fb75c005f90 comp rx=0 tx=0).stop 2024-09-17T05:27:36.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 msgr2=0x7fb77014d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb77014d1c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c380 tx=0x7fb764001d20 comp rx=0 tx=0).stop 2024-09-17T05:27:36.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 shutdown_connections 2024-09-17T05:27:36.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078b90 0x7fb75407b0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb77014a200 0x7fb77014a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb77014ba70 0x7fb7701f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 --2- 172.21.15.117:0/4064424202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb760003a00 0x7fb77014d1c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 >> 172.21.15.117:0/4064424202 conn(0x7fb7700744d0 msgr2=0x7fb770158500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:36.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 shutdown_connections 2024-09-17T05:27:36.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.721+0000 7fb776fed640 1 -- 172.21.15.117:0/4064424202 wait complete. 2024-09-17T05:27:36.735 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T05:27:36.735 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:36.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/3758433358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d41599b0 msgr2=0x7ff8d40c7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/3758433358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d41599b0 0x7ff8d40c7f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40cb470 tx=0x7ff8cc02aa30 comp rx=0 tx=0).stop 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/3758433358 shutdown_connections 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/3758433358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d41599b0 0x7ff8d40c7f60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/3758433358 >> v2:172.21.15.119:3300/0 conn(0x7ff8d415c5d0 0x7ff8d40c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/3758433358 >> 172.21.15.117:0/3758433358 conn(0x7ff8d40cd5e0 msgr2=0x7ff8d40cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/3758433358 shutdown_connections 2024-09-17T05:27:36.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.852+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/3758433358 wait complete. 2024-09-17T05:27:36.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.853+0000 7ff8dbc4d640 1 Processor -- start 2024-09-17T05:27:36.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.854+0000 7ff8dbc4d640 1 -- start start 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d4071090 0x7ff8d415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8d41599b0 0x7ff8d4147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 0x7ff8d414d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d4159dc0 con 0x7ff8d415c5d0 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d4159520 con 0x7ff8d4071090 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d40c7600 con 0x7ff8d41599b0 2024-09-17T05:27:36.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 0x7ff8d414d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 0x7ff8d414d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50534/0 (socket says 172.21.15.117:50534) 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 -- 172.21.15.117:0/184835713 learned_addr learned my addr 172.21.15.117:0/184835713 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8d91e8640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8d41599b0 0x7ff8d4147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8d99e9640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d4071090 0x7ff8d415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 -- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8d41599b0 msgr2=0x7ff8d4147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8d41599b0 0x7ff8d4147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 -- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d4071090 msgr2=0x7ff8d415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d4071090 0x7ff8d415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 -- 172.21.15.117:0/184835713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d415a890 con 0x7ff8d415c5d0 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8d91e8640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8d41599b0 0x7ff8d4147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8da1ea640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 0x7ff8d414d870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009bb0 tx=0x7ff8cc003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:36.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc033050 con 0x7ff8d415c5d0 2024-09-17T05:27:36.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8cc0044d0 con 0x7ff8d415c5d0 2024-09-17T05:27:36.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.855+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d41efcb0 con 0x7ff8d415c5d0 2024-09-17T05:27:36.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.856+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc038450 con 0x7ff8d415c5d0 2024-09-17T05:27:36.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.856+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d41f0550 con 0x7ff8d415c5d0 2024-09-17T05:27:36.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.857+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d40c37a0 con 0x7ff8d415c5d0 2024-09-17T05:27:36.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.857+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff8cc036070 con 0x7ff8d415c5d0 2024-09-17T05:27:36.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.858+0000 7ff8ca7fc640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff8b8078bc0 0x7ff8b807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:36.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.858+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff8cc0bc4c0 con 0x7ff8d415c5d0 2024-09-17T05:27:36.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.858+0000 7ff8d99e9640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff8b8078bc0 0x7ff8b807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:36.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.858+0000 7ff8d99e9640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff8b8078bc0 0x7ff8b807b0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00058c0 tx=0x7ff8d0005850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:36.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:36.862+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff8cc084650 con 0x7ff8d415c5d0 2024-09-17T05:27:37.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.071+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff8d4073e90 con 0x7ff8d415c5d0 2024-09-17T05:27:37.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.072+0000 7ff8ca7fc640 1 -- 172.21.15.117:0/184835713 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7ff8cc083f50 con 0x7ff8d415c5d0 2024-09-17T05:27:37.072 INFO:teuthology.orchestra.run.smithi117.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-17T05:27:37.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff8b8078bc0 msgr2=0x7ff8b807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:37.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff8b8078bc0 0x7ff8b807b0d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00058c0 tx=0x7ff8d0005850 comp rx=0 tx=0).stop 2024-09-17T05:27:37.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 msgr2=0x7ff8d414d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 0x7ff8d414d870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009bb0 tx=0x7ff8cc003f60 comp rx=0 tx=0).stop 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 shutdown_connections 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff8b8078bc0 0x7ff8b807b0d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8d415c5d0 0x7ff8d414d870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8d41599b0 0x7ff8d4147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 --2- 172.21.15.117:0/184835713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d4071090 0x7ff8d415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 >> 172.21.15.117:0/184835713 conn(0x7ff8d40cd5e0 msgr2=0x7ff8d40c7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 shutdown_connections 2024-09-17T05:27:37.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37.074+0000 7ff8dbc4d640 1 -- 172.21.15.117:0/184835713 wait complete. 2024-09-17T05:27:37.085 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T05:27:37.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T05:27:37.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi117.front.sepia.ceph.com 2024-09-17T05:27:37.085 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-17T05:27:37.085 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:37.086 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T05:27:37.086 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T05:27:37.086 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T05:27:37.086 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-17T05:27:37.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:37.086 DEBUG:teuthology.orchestra.run.smithi117:> ip addr 2024-09-17T05:27:37.101 INFO:teuthology.orchestra.run.smithi117.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T05:27:37.101 INFO:teuthology.orchestra.run.smithi117.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T05:27:37.101 INFO:teuthology.orchestra.run.smithi117.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 ::1/128 scope host 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8a brd ff:ff:ff:ff:ff:ff 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f0 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8b brd ff:ff:ff:ff:ff:ff 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f1 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:62 brd ff:ff:ff:ff:ff:ff 2024-09-17T05:27:37.102 INFO:teuthology.orchestra.run.smithi117.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T05:27:37.103 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:63 brd ff:ff:ff:ff:ff:ff 2024-09-17T05:27:37.103 INFO:teuthology.orchestra.run.smithi117.stdout: inet 172.21.15.117/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T05:27:37.103 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft 42782sec preferred_lft 42782sec 2024-09-17T05:27:37.103 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 fe80::ec4:7aff:fe8f:4463/64 scope link noprefixroute 2024-09-17T05:27:37.103 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-09-17T05:27:37.103 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add name ceph-brx type bridge 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr flush dev ceph-brx 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set ceph-brx up 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-17T05:27:37.104 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:27:37.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:27:37.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:37.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:37.299 DEBUG:teuthology.orchestra.run.smithi117:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T05:27:37.324 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-09-17T05:27:37.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:37.326 DEBUG:teuthology.orchestra.run.smithi117:> route 2024-09-17T05:27:37.390 INFO:teuthology.orchestra.run.smithi117.stdout:Kernel IP routing table 2024-09-17T05:27:37.390 INFO:teuthology.orchestra.run.smithi117.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T05:27:37.390 INFO:teuthology.orchestra.run.smithi117.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T05:27:37.390 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T05:27:37.390 INFO:teuthology.orchestra.run.smithi117.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T05:27:37.391 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:27:37.391 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:27:37.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T05:27:37.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T05:27:37.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T05:27:37.391 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:27:37.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:27:37.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:37.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:37.682 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-09-17T05:27:37.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:37.697 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list-id 2024-09-17T05:27:37.752 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:27:37.753 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:27:37.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:37.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-17T05:27:37.753 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:27:37.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:27:37.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:37.873 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-17T05:27:37.873 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:27:37.873 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:27:37.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-17T05:27:37.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-17T05:27:37.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-17T05:27:37.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-17T05:27:37.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-17T05:27:37.874 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:27:37.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:27:38.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:38.011 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:27:38.011 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:27:38.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set brx.0 up 2024-09-17T05:27:38.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set dev brx.0 master ceph-brx 2024-09-17T05:27:38.012 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:27:38.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:27:38.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:38.089 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-17T05:27:38.090 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T05:27:38.090 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:38.105 INFO:teuthology.orchestra.run.smithi117.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-17T05:27:38.105 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T05:27:38.105 DEBUG:teuthology.orchestra.run.smithi117:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:38.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe fuse 2024-09-17T05:27:38.227 DEBUG:teuthology.orchestra.run.smithi117:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T05:27:38.283 INFO:teuthology.orchestra.run.smithi117.stdout:/proc 2024-09-17T05:27:38.283 INFO:teuthology.orchestra.run.smithi117.stdout:/sys 2024-09-17T05:27:38.283 INFO:teuthology.orchestra.run.smithi117.stdout:/dev 2024-09-17T05:27:38.283 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/security 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/shm 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/pts 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/run 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/cgroup 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/pstore 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/bpf 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/ 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/selinux 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/hugepages 2024-09-17T05:27:38.284 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/mqueue 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/debug 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/tracing 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/fuse/connections 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/config 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-sysctl.service 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/nfs/rpc_pipefs 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/1000 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-0 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-1 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-2 2024-09-17T05:27:38.285 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-3 2024-09-17T05:27:38.286 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns 2024-09-17T05:27:38.286 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:38.286 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:38.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:38.286 DEBUG:teuthology.orchestra.run.smithi117:> ls /sys/fs/fuse/connections 2024-09-17T05:27:38.337 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-17T05:27:38.338 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-17T05:27:38.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe fuse 2024-09-17T05:27:38.404 DEBUG:teuthology.orchestra.run.smithi117:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/proc 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/sys 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/dev 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/security 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/shm 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/pts 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/run 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/cgroup 2024-09-17T05:27:38.461 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/pstore 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/bpf 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/ 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/selinux 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/hugepages 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/mqueue 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/debug 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/tracing 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/fuse/connections 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/config 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-sysctl.service 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-17T05:27:38.462 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/nfs/rpc_pipefs 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/1000 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-0 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-1 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-2 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-3 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:38.463 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:38.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:38.464 DEBUG:teuthology.orchestra.run.smithi117:> ls /sys/fs/fuse/connections 2024-09-17T05:27:38.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:2024-09-17T05:27:38.515+0000 7fc58c321480 -1 init, newargv = 0x556db755a3f0 newargc=15 2024-09-17T05:27:38.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:ceph-fuse[43010]: starting ceph client 2024-09-17T05:27:38.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:ceph-fuse[43010]: starting fuse 2024-09-17T05:27:39.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe fuse 2024-09-17T05:27:39.655 DEBUG:teuthology.orchestra.run.smithi117:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T05:27:39.714 INFO:teuthology.orchestra.run.smithi117.stdout:/proc 2024-09-17T05:27:39.714 INFO:teuthology.orchestra.run.smithi117.stdout:/sys 2024-09-17T05:27:39.714 INFO:teuthology.orchestra.run.smithi117.stdout:/dev 2024-09-17T05:27:39.714 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/security 2024-09-17T05:27:39.714 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/shm 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/pts 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/run 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/cgroup 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/pstore 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/bpf 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/ 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/selinux 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/hugepages 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/mqueue 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/debug 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/tracing 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/fuse/connections 2024-09-17T05:27:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/config 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-sysctl.service 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/nfs/rpc_pipefs 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/1000 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-0 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-1 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-2 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph/osd/ceph-3 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:39.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:27:39.717 INFO:teuthology.orchestra.run.smithi117.stdout:/home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:39.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:39.718 DEBUG:teuthology.orchestra.run.smithi117:> ls /sys/fs/fuse/connections 2024-09-17T05:27:39.776 INFO:teuthology.orchestra.run.smithi117.stdout:41 2024-09-17T05:27:39.776 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo stdin-killer -- python3 -c ' 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> import glob 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> import re 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> import os 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> import subprocess 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> def _find_admin_socket(client_name): 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> files = glob.glob(asok_path) 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> # Given a non-glob path, it better be there 2024-09-17T05:27:39.777 DEBUG:teuthology.orchestra.run.smithi117:> if "*" not in asok_path: 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> assert(len(files) == 1) 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> return files[0] 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> for f in files: 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> contents = proc_f.read() 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> if mountpoint in contents: 2024-09-17T05:27:39.778 DEBUG:teuthology.orchestra.run.smithi117:> return f 2024-09-17T05:27:39.779 DEBUG:teuthology.orchestra.run.smithi117:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-17T05:27:39.779 DEBUG:teuthology.orchestra.run.smithi117:> 2024-09-17T05:27:39.779 DEBUG:teuthology.orchestra.run.smithi117:> print(_find_admin_socket("client.admin")) 2024-09-17T05:27:39.779 DEBUG:teuthology.orchestra.run.smithi117:> ' 2024-09-17T05:27:39.902 INFO:teuthology.orchestra.run.smithi117.stdout:/var/run/ceph/ceph-client.admin.43010.asok 2024-09-17T05:27:39.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:39.914 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43010.asok 2024-09-17T05:27:39.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:39.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43010.asok status 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout: "metadata": { 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_sha1": "ce1b342bf8c6349a8c90cd607f83bff2443f21cf", 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.3.0-4989-gce1b342b (ce1b342bf8c6349a8c90cd607f83bff2443f21cf) squid (dev)", 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout: "entity_id": "admin", 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout: "hostname": "smithi117", 2024-09-17T05:27:40.057 INFO:teuthology.orchestra.run.smithi117.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "pid": "43010", 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "root": "/" 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "dentry_count": 0, 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "dentry_pinned_count": 0, 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 4563, 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "inst": { 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "name": { 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "client", 2024-09-17T05:27:40.058 INFO:teuthology.orchestra.run.smithi117.stdout: "num": 4563 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": { 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v1", 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "192.168.0.1:0", 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 3834264173 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": { 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v1", 2024-09-17T05:27:40.059 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "192.168.0.1:0", 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 3834264173 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "inst_str": "client.4563 192.168.0.1:0/3834264173", 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "addr_str": "192.168.0.1:0/3834264173", 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "inode_count": 1, 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_epoch": 5, 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_epoch": 25, 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_epoch_barrier": 0, 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "blocklisted": false, 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs" 2024-09-17T05:27:40.060 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:27:40.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:40.066 DEBUG:teuthology.orchestra.run.smithi117:> stat --file-system '--printf=%T 2024-09-17T05:27:40.066 DEBUG:teuthology.orchestra.run.smithi117:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:40.082 INFO:teuthology.orchestra.run.smithi117.stdout:fuseblk 2024-09-17T05:27:40.082 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:40.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:40.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:40.148 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-17T05:27:40.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T05:27:40.253 INFO:teuthology.orchestra.run.smithi117.stdout:# file: . 2024-09-17T05:27:40.253 INFO:teuthology.orchestra.run.smithi117.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-17T05:27:40.253 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:40.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:27:40.259 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:27:40.259 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:27:40.275 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1017303954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac0c9af0 msgr2=0x7fd0ac159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1017303954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac0c9af0 0x7fd0ac159a50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009830 tx=0x7fd09c02a5a0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1017303954 shutdown_connections 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1017303954 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0ac06cda0 0x7fd0ac06d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1017303954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac0c9af0 0x7fd0ac159a50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1017303954 >> 172.21.15.117:0/1017303954 conn(0x7fd0ac0ccbe0 msgr2=0x7fd0ac0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1017303954 shutdown_connections 2024-09-17T05:27:40.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.427+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1017303954 wait complete. 2024-09-17T05:27:40.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.429+0000 7fd0b0eb4640 1 Processor -- start 2024-09-17T05:27:40.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.429+0000 7fd0b0eb4640 1 -- start start 2024-09-17T05:27:40.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0b0eb4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0ac06cda0 0x7fd0ac1563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0b0eb4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0ac0c9af0 0x7fd0ac1f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0b0eb4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 0x7fd0ac074410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0b0eb4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac0c76b0 con 0x7fd0ac14fb50 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0b0eb4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac0c73d0 con 0x7fd0ac06cda0 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0b0eb4640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac1535e0 con 0x7fd0ac0c9af0 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 0x7fd0ac074410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 0x7fd0ac074410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50552/0 (socket says 172.21.15.117:50552) 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.430+0000 7fd0aad9d640 1 -- 172.21.15.117:0/1333142215 learned_addr learned my addr 172.21.15.117:0/1333142215 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aa59c640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0ac06cda0 0x7fd0ac1563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0a9d9b640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0ac0c9af0 0x7fd0ac1f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aad9d640 1 -- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0ac0c9af0 msgr2=0x7fd0ac1f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aad9d640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0ac0c9af0 0x7fd0ac1f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aad9d640 1 -- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0ac06cda0 msgr2=0x7fd0ac1563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aad9d640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0ac06cda0 0x7fd0ac1563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aad9d640 1 -- 172.21.15.117:0/1333142215 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac1555a0 con 0x7fd0ac14fb50 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0aad9d640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 0x7fd0ac074410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400e960 tx=0x7fd0a400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4004d20 con 0x7fd0ac14fb50 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0a400cb20 con 0x7fd0ac14fb50 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac1e23e0 con 0x7fd0ac14fb50 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4005020 con 0x7fd0ac14fb50 2024-09-17T05:27:40.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.431+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ac1e36b0 con 0x7fd0ac14fb50 2024-09-17T05:27:40.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.432+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ac15a190 con 0x7fd0ac14fb50 2024-09-17T05:27:40.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.433+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd0a40051c0 con 0x7fd0ac14fb50 2024-09-17T05:27:40.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.433+0000 7fd09affd640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd08c078bc0 0x7fd08c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.434+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd0a409ba10 con 0x7fd0ac14fb50 2024-09-17T05:27:40.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.434+0000 7fd0aa59c640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd08c078bc0 0x7fd08c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.434+0000 7fd0aa59c640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd08c078bc0 0x7fd08c07b0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd09c034000 tx=0x7fd09c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:40.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.438+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0ac15a190 con 0x7fd0ac14fb50 2024-09-17T05:27:40.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.625+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd0ac15a4c0 con 0x7fd0ac14fb50 2024-09-17T05:27:40.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.625+0000 7fd09affd640 1 -- 172.21.15.117:0/1333142215 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7fd0a4016030 con 0x7fd0ac14fb50 2024-09-17T05:27:40.626 INFO:teuthology.orchestra.run.smithi117.stderr:listed 0 entries 2024-09-17T05:27:40.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.627+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd08c078bc0 msgr2=0x7fd08c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.627+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd08c078bc0 0x7fd08c07b0d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd09c034000 tx=0x7fd09c004730 comp rx=0 tx=0).stop 2024-09-17T05:27:40.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 msgr2=0x7fd0ac074410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 0x7fd0ac074410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400e960 tx=0x7fd0a400ee20 comp rx=0 tx=0).stop 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 shutdown_connections 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd08c078bc0 0x7fd08c07b0d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0ac14fb50 0x7fd0ac074410 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0ac0c9af0 0x7fd0ac1f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 --2- 172.21.15.117:0/1333142215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0ac06cda0 0x7fd0ac1563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 >> 172.21.15.117:0/1333142215 conn(0x7fd0ac0ccbe0 msgr2=0x7fd0ac14f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 shutdown_connections 2024-09-17T05:27:40.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.628+0000 7fd0b0eb4640 1 -- 172.21.15.117:0/1333142215 wait complete. 2024-09-17T05:27:40.639 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:27:40.639 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:27:40.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 -- 172.21.15.117:0/2846698197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c0cbbd0 msgr2=0x7f390c151ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 --2- 172.21.15.117:0/2846698197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c0cbbd0 0x7f390c151ec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3908005fd0 tx=0x7f390802a670 comp rx=0 tx=0).stop 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 -- 172.21.15.117:0/2846698197 shutdown_connections 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 --2- 172.21.15.117:0/2846698197 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f390c15bdc0 0x7f390c0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 --2- 172.21.15.117:0/2846698197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c0cbbd0 0x7f390c151ec0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 -- 172.21.15.117:0/2846698197 >> 172.21.15.117:0/2846698197 conn(0x7f390c06e170 msgr2=0x7f390c06e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 -- 172.21.15.117:0/2846698197 shutdown_connections 2024-09-17T05:27:40.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.807+0000 7f3913d6e640 1 -- 172.21.15.117:0/2846698197 wait complete. 2024-09-17T05:27:40.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.809+0000 7f3913d6e640 1 Processor -- start 2024-09-17T05:27:40.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.809+0000 7f3913d6e640 1 -- start start 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3913d6e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f390c0b4db0 0x7f390c1f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3913d6e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f390c0cbbd0 0x7f390c152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3913d6e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 0x7f390c15d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3913d6e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f390c15f090 con 0x7f390c15bdc0 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3913d6e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f390c15edb0 con 0x7f390c0cbbd0 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3913d6e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f390c0b43e0 con 0x7f390c0b4db0 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 0x7f390c15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 0x7f390c15d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50576/0 (socket says 172.21.15.117:50576) 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 -- 172.21.15.117:0/52713483 learned_addr learned my addr 172.21.15.117:0/52713483 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:40.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3911309640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f390c0cbbd0 0x7f390c152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3911b0a640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f390c0b4db0 0x7f390c1f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 -- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f390c0b4db0 msgr2=0x7f390c1f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f390c0b4db0 0x7f390c1f5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 -- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f390c0cbbd0 msgr2=0x7f390c152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f390c0cbbd0 0x7f390c152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f391230b640 1 -- 172.21.15.117:0/52713483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c0ce890 con 0x7f390c15bdc0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.810+0000 7f3911309640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f390c0cbbd0 0x7f390c152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.811+0000 7f391230b640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 0x7f390c15d860 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f390400c6b0 tx=0x7f390400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.811+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904003b70 con 0x7f390c15bdc0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.811+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3904003d30 con 0x7f390c15bdc0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.811+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904005330 con 0x7f390c15bdc0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.811+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390c0ceb70 con 0x7f390c15bdc0 2024-09-17T05:27:40.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.811+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f390c153a50 con 0x7f390c15bdc0 2024-09-17T05:27:40.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.812+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f390c15c1d0 con 0x7f390c15bdc0 2024-09-17T05:27:40.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.813+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f39040054d0 con 0x7f390c15bdc0 2024-09-17T05:27:40.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.813+0000 7f39027fc640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38f0078bc0 0x7f38f007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:40.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.813+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f390409add0 con 0x7f390c15bdc0 2024-09-17T05:27:40.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.813+0000 7f3911b0a640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38f0078bc0 0x7f38f007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:40.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.814+0000 7f3911b0a640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38f0078bc0 0x7f38f007b0d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3908033000 tx=0x7f3908004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:40.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:40.818+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f390c0c7610 con 0x7f390c15bdc0 2024-09-17T05:27:41.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.003+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f390c0cce60 con 0x7f390c15bdc0 2024-09-17T05:27:41.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.003+0000 7f39027fc640 1 -- 172.21.15.117:0/52713483 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f3904062f60 con 0x7f390c15bdc0 2024-09-17T05:27:41.004 INFO:teuthology.orchestra.run.smithi117.stderr:listed 0 entries 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38f0078bc0 msgr2=0x7f38f007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38f0078bc0 0x7f38f007b0d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3908033000 tx=0x7f3908004730 comp rx=0 tx=0).stop 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 msgr2=0x7f390c15d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 0x7f390c15d860 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f390400c6b0 tx=0x7f390400cb70 comp rx=0 tx=0).stop 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 shutdown_connections 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38f0078bc0 0x7f38f007b0d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390c15bdc0 0x7f390c15d860 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f390c0cbbd0 0x7f390c152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 --2- 172.21.15.117:0/52713483 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f390c0b4db0 0x7f390c1f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 >> 172.21.15.117:0/52713483 conn(0x7f390c06e170 msgr2=0x7f390c0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:41.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 shutdown_connections 2024-09-17T05:27:41.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.005+0000 7f3913d6e640 1 -- 172.21.15.117:0/52713483 wait complete. 2024-09-17T05:27:41.017 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi117.front.sepia.ceph.com... 2024-09-17T05:27:41.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:41.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-17T05:27:41.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:ceph-fuse[43010]: fuse finished with error 0 and tester_r 0 2024-09-17T05:27:41.042 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:27:41.093 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:27:41.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:27:41.093 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-17T05:27:41.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:41.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 -- 172.21.15.117:0/2766144222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500cb390 msgr2=0x7f59500d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 --2- 172.21.15.117:0/2766144222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500cb390 0x7f59500d08f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5940009830 tx=0x7f594002a640 comp rx=0 tx=0).stop 2024-09-17T05:27:41.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 -- 172.21.15.117:0/2766144222 shutdown_connections 2024-09-17T05:27:41.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 --2- 172.21.15.117:0/2766144222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500cb390 0x7f59500d08f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 -- 172.21.15.117:0/2766144222 >> 172.21.15.117:0/2766144222 conn(0x7f5950075e00 msgr2=0x7f595006e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:41.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 -- 172.21.15.117:0/2766144222 shutdown_connections 2024-09-17T05:27:41.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.268+0000 7f5956739640 1 -- 172.21.15.117:0/2766144222 wait complete. 2024-09-17T05:27:41.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.270+0000 7f5956739640 1 Processor -- start 2024-09-17T05:27:41.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.270+0000 7f5956739640 1 -- start start 2024-09-17T05:27:41.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f5956739640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 0x7f59501f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f5956739640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59500cb390 0x7f59500c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f5956739640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f595015b2f0 0x7f59500c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f5956739640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f59500c5da0 con 0x7f59500c69d0 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f5956739640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f595015f6b0 con 0x7f595015b2f0 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f5956739640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f59501558b0 con 0x7f59500cb390 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f594ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 0x7f59501f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f594ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 0x7f59501f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50592/0 (socket says 172.21.15.117:50592) 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.271+0000 7f594ffff640 1 -- 172.21.15.117:0/1649610366 learned_addr learned my addr 172.21.15.117:0/1649610366 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594f7fe640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59500cb390 0x7f59500c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f5954cd6640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f595015b2f0 0x7f59500c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594ffff640 1 -- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59500cb390 msgr2=0x7f59500c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594ffff640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59500cb390 0x7f59500c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594ffff640 1 -- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f595015b2f0 msgr2=0x7f59500c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594ffff640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f595015b2f0 0x7f59500c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594ffff640 1 -- 172.21.15.117:0/1649610366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59500cafb0 con 0x7f59500c69d0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594ffff640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 0x7f59501f1a90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5940000c00 tx=0x7f5940033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59400332e0 con 0x7f59500c69d0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5940033570 con 0x7f59500c69d0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59501f1fd0 con 0x7f59500c69d0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940031a10 con 0x7f59500c69d0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.272+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59500d0880 con 0x7f59500c69d0 2024-09-17T05:27:41.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.273+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59500c5b80 con 0x7f59500c69d0 2024-09-17T05:27:41.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.274+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5940031bb0 con 0x7f59500c69d0 2024-09-17T05:27:41.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.274+0000 7f594cff9640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f593c078b90 0x7f593c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.275+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f59400bc320 con 0x7f59500c69d0 2024-09-17T05:27:41.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.275+0000 7f594f7fe640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f593c078b90 0x7f593c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.275+0000 7f594f7fe640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f593c078b90 0x7f593c07b0a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59500b5120 tx=0x7f5944007eb0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:41.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.279+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f59500c5b80 con 0x7f59500c69d0 2024-09-17T05:27:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.455+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59500c7950 con 0x7f59500c69d0 2024-09-17T05:27:41.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.456+0000 7f594cff9640 1 -- 172.21.15.117:0/1649610366 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f594003f030 con 0x7f59500c69d0 2024-09-17T05:27:41.456 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:41.456 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:41.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f593c078b90 msgr2=0x7f593c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f593c078b90 0x7f593c07b0a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59500b5120 tx=0x7f5944007eb0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 msgr2=0x7f59501f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 0x7f59501f1a90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5940000c00 tx=0x7f5940033df0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 shutdown_connections 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f593c078b90 0x7f593c07b0a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f595015b2f0 0x7f59500c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59500cb390 0x7f59500c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 --2- 172.21.15.117:0/1649610366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59500c69d0 0x7f59501f1a90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 >> 172.21.15.117:0/1649610366 conn(0x7f5950075e00 msgr2=0x7f5950158f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 shutdown_connections 2024-09-17T05:27:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.458+0000 7f5956739640 1 -- 172.21.15.117:0/1649610366 wait complete. 2024-09-17T05:27:41.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 -- 172.21.15.117:0/587683921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100c35d0 msgr2=0x7f9c100c39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 --2- 172.21.15.117:0/587683921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100c35d0 0x7f9c100c39e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9c100723f0 tx=0x7f9c0802aa30 comp rx=0 tx=0).stop 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 -- 172.21.15.117:0/587683921 shutdown_connections 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 --2- 172.21.15.117:0/587683921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100c35d0 0x7f9c100c39e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 --2- 172.21.15.117:0/587683921 >> v2:172.21.15.119:3300/0 conn(0x7f9c100cb080 0x7f9c100cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 --2- 172.21.15.117:0/587683921 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9c10153210 0x7f9c10159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 -- 172.21.15.117:0/587683921 >> 172.21.15.117:0/587683921 conn(0x7f9c10072cc0 msgr2=0x7f9c100730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 -- 172.21.15.117:0/587683921 shutdown_connections 2024-09-17T05:27:41.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.582+0000 7f9c17864640 1 -- 172.21.15.117:0/587683921 wait complete. 2024-09-17T05:27:41.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.584+0000 7f9c17864640 1 Processor -- start 2024-09-17T05:27:41.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.584+0000 7f9c17864640 1 -- start start 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c17864640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100c35d0 0x7f9c10151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c17864640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 0x7f9c10158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c17864640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9c10153210 0x7f9c10145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c17864640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1015d0d0 con 0x7f9c100cb080 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c17864640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1015cdf0 con 0x7f9c100c35d0 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c17864640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f9c100c95f0 con 0x7f9c10153210 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c14dff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 0x7f9c10158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c14dff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 0x7f9c10158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50606/0 (socket says 172.21.15.117:50606) 2024-09-17T05:27:41.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.585+0000 7f9c14dff640 1 -- 172.21.15.117:0/304868863 learned_addr learned my addr 172.21.15.117:0/304868863 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c15600640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100c35d0 0x7f9c10151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c14dff640 1 -- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9c10153210 msgr2=0x7f9c10145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c14dff640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9c10153210 0x7f9c10145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c14dff640 1 -- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100c35d0 msgr2=0x7f9c10151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c14dff640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100c35d0 0x7f9c10151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c14dff640 1 -- 172.21.15.117:0/304868863 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c101482c0 con 0x7f9c100cb080 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c14dff640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 0x7f9c10158580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000d680 tx=0x7f9c0000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00017070 con 0x7f9c100cb080 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c00004310 con 0x7f9c100cb080 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1014dd70 con 0x7f9c100cb080 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00005280 con 0x7f9c100cb080 2024-09-17T05:27:41.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.586+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c10157e00 con 0x7f9c100cb080 2024-09-17T05:27:41.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.587+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c10159a90 con 0x7f9c100cb080 2024-09-17T05:27:41.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.588+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9c00005420 con 0x7f9c100cb080 2024-09-17T05:27:41.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.588+0000 7f9c05ffb640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9bf4078b90 0x7f9bf407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.588+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9c0009b350 con 0x7f9c100cb080 2024-09-17T05:27:41.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.589+0000 7f9c15600640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9bf4078b90 0x7f9bf407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.589+0000 7f9c15600640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9bf4078b90 0x7f9bf407b0a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005d00 tx=0x7f9c0c005c70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:41.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.593+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9c00020020 con 0x7f9c100cb080 2024-09-17T05:27:41.626 INFO:tasks.ceph.mds.f.smithi119.stderr:2024-09-17T05:27:41.626+0000 7f6f66926640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:27:41.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.773+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c100ca1a0 con 0x7f9c100cb080 2024-09-17T05:27:41.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.774+0000 7f9c05ffb640 1 -- 172.21.15.117:0/304868863 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f9c00063570 con 0x7f9c100cb080 2024-09-17T05:27:41.774 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:41.775 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9bf4078b90 msgr2=0x7f9bf407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9bf4078b90 0x7f9bf407b0a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005d00 tx=0x7f9c0c005c70 comp rx=0 tx=0).stop 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 msgr2=0x7f9c10158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 0x7f9c10158580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9c0000d680 tx=0x7f9c0000db40 comp rx=0 tx=0).stop 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 shutdown_connections 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9bf4078b90 0x7f9bf407b0a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9c10153210 0x7f9c10145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c100cb080 0x7f9c10158580 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 --2- 172.21.15.117:0/304868863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100c35d0 0x7f9c10151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 >> 172.21.15.117:0/304868863 conn(0x7f9c10072cc0 msgr2=0x7f9c100b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:41.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 shutdown_connections 2024-09-17T05:27:41.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.776+0000 7f9c17864640 1 -- 172.21.15.117:0/304868863 wait complete. 2024-09-17T05:27:41.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T05:27:41.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6d009ad640 1 -- 172.21.15.117:0/1671708112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce4004d40 msgr2=0x7f6ce4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1671708112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce4004d40 0x7f6ce4005150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0c9670 tx=0x7f6ce4005830 comp rx=0 tx=0).stop 2024-09-17T05:27:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6cfad9d640 1 -- 172.21.15.117:0/1671708112 reap_dead start 2024-09-17T05:27:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6d009ad640 1 -- 172.21.15.117:0/1671708112 shutdown_connections 2024-09-17T05:27:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6d009ad640 1 -- 172.21.15.117:0/1671708112 >> 172.21.15.117:0/1671708112 conn(0x7f6cfc076430 msgr2=0x7f6cfc072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6d009ad640 1 -- 172.21.15.117:0/1671708112 shutdown_connections 2024-09-17T05:27:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.903+0000 7f6d009ad640 1 -- 172.21.15.117:0/1671708112 wait complete. 2024-09-17T05:27:41.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.905+0000 7f6d009ad640 1 Processor -- start 2024-09-17T05:27:41.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.905+0000 7f6d009ad640 1 -- start start 2024-09-17T05:27:41.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6d009ad640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce4004d40 0x7f6cfc0b2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6d009ad640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 0x7f6cfc15a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6d009ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cfc14ee10 0x7f6cfc145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6d009ad640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc0cbc30 con 0x7f6cfc0b3950 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6d009ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc15cd00 con 0x7f6cfc14ee10 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6d009ad640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6cfc0cb230 con 0x7f6ce4004d40 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 0x7f6cfc15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 0x7f6cfc15a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50622/0 (socket says 172.21.15.117:50622) 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 -- 172.21.15.117:0/1569629961 learned_addr learned my addr 172.21.15.117:0/1569629961 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cfad9d640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cfc14ee10 0x7f6cfc145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cfa59c640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce4004d40 0x7f6cfc0b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 -- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce4004d40 msgr2=0x7f6cfc0b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce4004d40 0x7f6cfc0b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 -- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cfc14ee10 msgr2=0x7f6cfc145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cfc14ee10 0x7f6cfc145fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 -- 172.21.15.117:0/1569629961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc0b2ed0 con 0x7f6cfc0b3950 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.906+0000 7f6cf9d9b640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 0x7f6cfc15a270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0011ae0 tx=0x7f6cf0011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.907+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0004440 con 0x7f6cfc0b3950 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.907+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cf000f9a0 con 0x7f6cfc0b3950 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.907+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc15a7b0 con 0x7f6cfc0b3950 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.907+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000fdd0 con 0x7f6cfc0b3950 2024-09-17T05:27:41.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.907+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc15bf90 con 0x7f6cfc0b3950 2024-09-17T05:27:41.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.908+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cfc0c37c0 con 0x7f6cfc0b3950 2024-09-17T05:27:41.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.908+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6cf00035f0 con 0x7f6cfc0b3950 2024-09-17T05:27:41.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.909+0000 7f6ceaffd640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cdc078b90 0x7f6cdc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:41.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.909+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6cf009c060 con 0x7f6cfc0b3950 2024-09-17T05:27:41.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.909+0000 7f6cfa59c640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cdc078b90 0x7f6cdc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:41.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.910+0000 7f6cfa59c640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cdc078b90 0x7f6cdc07b0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006d10 tx=0x7f6cec0067a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:41.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:41.913+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6cf00641f0 con 0x7f6cfc0b3950 2024-09-17T05:27:42.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.124+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6cfc0c4900 con 0x7f6cfc0b3950 2024-09-17T05:27:42.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.713+0000 7f6ceaffd640 1 -- 172.21.15.117:0/1569629961 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f6cf0063af0 con 0x7f6cfc0b3950 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cdc078b90 msgr2=0x7f6cdc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cdc078b90 0x7f6cdc07b0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006d10 tx=0x7f6cec0067a0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 msgr2=0x7f6cfc15a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 0x7f6cfc15a270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0011ae0 tx=0x7f6cf0011ee0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 shutdown_connections 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cdc078b90 0x7f6cdc07b0a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cfc14ee10 0x7f6cfc145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cfc0b3950 0x7f6cfc15a270 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 --2- 172.21.15.117:0/1569629961 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce4004d40 0x7f6cfc0b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 >> 172.21.15.117:0/1569629961 conn(0x7f6cfc076430 msgr2=0x7f6cfc148290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:42.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 shutdown_connections 2024-09-17T05:27:42.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.716+0000 7f6d009ad640 1 -- 172.21.15.117:0/1569629961 wait complete. 2024-09-17T05:27:42.729 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300} 2024-09-17T05:27:42.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-09-17T05:27:42.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.846+0000 7f5ded6b6640 1 -- 172.21.15.117:0/1222027127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de815d220 msgr2=0x7f5de80c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:42.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.846+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/1222027127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de815d220 0x7f5de80c9730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5de0005fd0 tx=0x7f5de002aa90 comp rx=0 tx=0).stop 2024-09-17T05:27:42.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.846+0000 7f5ded6b6640 1 -- 172.21.15.117:0/1222027127 shutdown_connections 2024-09-17T05:27:42.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.846+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/1222027127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de815d220 0x7f5de80c9730 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.846+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/1222027127 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5de8153260 0x7f5de80c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.846+0000 7f5ded6b6640 1 -- 172.21.15.117:0/1222027127 >> 172.21.15.117:0/1222027127 conn(0x7f5de8076430 msgr2=0x7f5de8072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:42.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.847+0000 7f5ded6b6640 1 -- 172.21.15.117:0/1222027127 shutdown_connections 2024-09-17T05:27:42.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.847+0000 7f5ded6b6640 1 -- 172.21.15.117:0/1222027127 wait complete. 2024-09-17T05:27:42.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.848+0000 7f5ded6b6640 1 Processor -- start 2024-09-17T05:27:42.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.848+0000 7f5ded6b6640 1 -- start start 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.849+0000 7f5ded6b6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5de80cad30 0x7f5de814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.849+0000 7f5ded6b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 0x7f5de815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.849+0000 7f5ded6b6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5de815d220 0x7f5de81460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ded6b6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5de80cbc30 con 0x7f5de8153260 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ded6b6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5de815cd00 con 0x7f5de80cad30 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ded6b6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5de80cb230 con 0x7f5de815d220 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 0x7f5de815b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 0x7f5de815b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50644/0 (socket says 172.21.15.117:50644) 2024-09-17T05:27:42.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 -- 172.21.15.117:0/921239554 learned_addr learned my addr 172.21.15.117:0/921239554 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5de6ffd640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5de80cad30 0x7f5de814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5de77fe640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5de815d220 0x7f5de81460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 -- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5de815d220 msgr2=0x7f5de81460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5de815d220 0x7f5de81460b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 -- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5de80cad30 msgr2=0x7f5de814a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5de80cad30 0x7f5de814a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 -- 172.21.15.117:0/921239554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de814a830 con 0x7f5de8153260 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddffff640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 0x7f5de815b010 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000d6d0 tx=0x7f5dd000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0017070 con 0x7f5de8153260 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dd0005330 con 0x7f5de8153260 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de814c2e0 con 0x7f5de8153260 2024-09-17T05:27:42.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0009820 con 0x7f5de8153260 2024-09-17T05:27:42.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.850+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de814c880 con 0x7f5de8153260 2024-09-17T05:27:42.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.851+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de80c37c0 con 0x7f5de8153260 2024-09-17T05:27:42.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.852+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5dd0009bd0 con 0x7f5de8153260 2024-09-17T05:27:42.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.852+0000 7f5ddf7fe640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5dc8078b90 0x7f5dc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:42.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.853+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5dd009b210 con 0x7f5de8153260 2024-09-17T05:27:42.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.853+0000 7f5de6ffd640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5dc8078b90 0x7f5dc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:42.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.853+0000 7f5de6ffd640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5dc8078b90 0x7f5dc807b0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8001720 tx=0x7f5dd8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:42.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:42.857+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5dd0062de0 con 0x7f5de8153260 2024-09-17T05:27:43.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.065+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f5de80ca300 con 0x7f5de8153260 2024-09-17T05:27:43.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.726+0000 7f5ddf7fe640 1 -- 172.21.15.117:0/921239554 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v8) ==== 105+0+0 (secure 0 0 0) 0x7f5dd0067810 con 0x7f5de8153260 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5dc8078b90 msgr2=0x7f5dc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5dc8078b90 0x7f5dc807b0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8001720 tx=0x7f5dd8009040 comp rx=0 tx=0).stop 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 msgr2=0x7f5de815b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 0x7f5de815b010 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000d6d0 tx=0x7f5dd000db90 comp rx=0 tx=0).stop 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 shutdown_connections 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5dc8078b90 0x7f5dc807b0a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:43.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5de815d220 0x7f5de81460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:43.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5de8153260 0x7f5de815b010 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:43.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 --2- 172.21.15.117:0/921239554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5de80cad30 0x7f5de814a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:43.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 >> 172.21.15.117:0/921239554 conn(0x7f5de8076430 msgr2=0x7f5de8156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:43.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.729+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 shutdown_connections 2024-09-17T05:27:43.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.730+0000 7f5ded6b6640 1 -- 172.21.15.117:0/921239554 wait complete. 2024-09-17T05:27:43.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T05:27:43.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb16f72640 1 -- 172.21.15.117:0/3963808912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaf8004800 msgr2=0x7feaf8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:43.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb16f72640 1 --2- 172.21.15.117:0/3963808912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaf8004800 0x7feaf8004c10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb10074a80 tx=0x7feaf8005830 comp rx=0 tx=0).stop 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb1550f640 1 -- 172.21.15.117:0/3963808912 reap_dead start 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb16f72640 1 -- 172.21.15.117:0/3963808912 shutdown_connections 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb16f72640 1 -- 172.21.15.117:0/3963808912 >> 172.21.15.117:0/3963808912 conn(0x7feb1006d3d0 msgr2=0x7feb10072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb16f72640 1 -- 172.21.15.117:0/3963808912 shutdown_connections 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.866+0000 7feb16f72640 1 -- 172.21.15.117:0/3963808912 wait complete. 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.867+0000 7feb16f72640 1 Processor -- start 2024-09-17T05:27:43.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.867+0000 7feb16f72640 1 -- start start 2024-09-17T05:27:43.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb16f72640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feaf8004800 0x7feb100c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:43.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb16f72640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 0x7feb101460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:43.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb16f72640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb1015a2c0 0x7feb101ef5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:43.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb16f72640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feb10159a70 con 0x7feb101466a0 2024-09-17T05:27:43.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb16f72640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feb10159790 con 0x7feb1015a2c0 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb16f72640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7feb10159d50 con 0x7feaf8004800 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 0x7feb101460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 0x7feb101460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50660/0 (socket says 172.21.15.117:50660) 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 -- 172.21.15.117:0/4255494299 learned_addr learned my addr 172.21.15.117:0/4255494299 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb1550f640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb1015a2c0 0x7feb101ef5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb14d0e640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feaf8004800 0x7feb100c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 -- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feaf8004800 msgr2=0x7feb100c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feaf8004800 0x7feb100c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 -- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb1015a2c0 msgr2=0x7feb101ef5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb1015a2c0 0x7feb101ef5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 -- 172.21.15.117:0/4255494299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb101efb30 con 0x7feb101466a0 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.869+0000 7feb07fff640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 0x7feb101460a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7feb00009a70 tx=0x7feb00033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:43.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.870+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00043070 con 0x7feb101466a0 2024-09-17T05:27:43.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.870+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb00004140 con 0x7feb101466a0 2024-09-17T05:27:43.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.870+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb10149850 con 0x7feb101466a0 2024-09-17T05:27:43.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.870+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00038040 con 0x7feb101466a0 2024-09-17T05:27:43.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.870+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb100c5c70 con 0x7feb101466a0 2024-09-17T05:27:43.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.871+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb100c65a0 con 0x7feb101466a0 2024-09-17T05:27:43.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.871+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feb00046070 con 0x7feb101466a0 2024-09-17T05:27:43.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.872+0000 7feb057fa640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feafc078b90 0x7feafc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:43.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.872+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7feb000bb5d0 con 0x7feb101466a0 2024-09-17T05:27:43.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.872+0000 7feb14d0e640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feafc078b90 0x7feafc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:43.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.873+0000 7feb14d0e640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feafc078b90 0x7feafc07b0a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009b30 tx=0x7feb0c005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:43.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:43.876+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feb00083110 con 0x7feb101466a0 2024-09-17T05:27:44.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.087+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7feb100c6390 con 0x7feb101466a0 2024-09-17T05:27:44.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.734+0000 7feb057fa640 1 -- 172.21.15.117:0/4255494299 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7feb00087b40 con 0x7feb101466a0 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feafc078b90 msgr2=0x7feafc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feafc078b90 0x7feafc07b0a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009b30 tx=0x7feb0c005f90 comp rx=0 tx=0).stop 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 msgr2=0x7feb101460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 0x7feb101460a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7feb00009a70 tx=0x7feb00033670 comp rx=0 tx=0).stop 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 shutdown_connections 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feafc078b90 0x7feafc07b0a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb1015a2c0 0x7feb101ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb101466a0 0x7feb101460a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 --2- 172.21.15.117:0/4255494299 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feaf8004800 0x7feb100c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 >> 172.21.15.117:0/4255494299 conn(0x7feb1006d3d0 msgr2=0x7feb10157630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 shutdown_connections 2024-09-17T05:27:44.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.737+0000 7feb16f72640 1 -- 172.21.15.117:0/4255494299 wait complete. 2024-09-17T05:27:44.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-09-17T05:27:44.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 -- 172.21.15.117:0/2823757964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c0cbbd0 msgr2=0x7f637c151ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:44.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 --2- 172.21.15.117:0/2823757964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c0cbbd0 0x7f637c151ec0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6374005fd0 tx=0x7f637402aa90 comp rx=0 tx=0).stop 2024-09-17T05:27:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 -- 172.21.15.117:0/2823757964 shutdown_connections 2024-09-17T05:27:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 --2- 172.21.15.117:0/2823757964 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f637c0cd290 0x7f637c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 --2- 172.21.15.117:0/2823757964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c0cbbd0 0x7f637c151ec0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 -- 172.21.15.117:0/2823757964 >> 172.21.15.117:0/2823757964 conn(0x7f637c06e170 msgr2=0x7f637c06e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 -- 172.21.15.117:0/2823757964 shutdown_connections 2024-09-17T05:27:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.871+0000 7f6380ccc640 1 -- 172.21.15.117:0/2823757964 wait complete. 2024-09-17T05:27:44.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.873+0000 7f6380ccc640 1 Processor -- start 2024-09-17T05:27:44.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.873+0000 7f6380ccc640 1 -- start start 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6380ccc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f637c0cbbd0 0x7f637c1f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6380ccc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f637c0cd290 0x7f637c0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6380ccc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 0x7f637c0cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6380ccc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f637c15f090 con 0x7f637c15bdc0 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6380ccc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f637c15edb0 con 0x7f637c0cd290 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6380ccc640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f637c0b43e0 con 0x7f637c0cbbd0 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 0x7f637c0cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 0x7f637c0cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39906/0 (socket says 172.21.15.117:39906) 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 -- 172.21.15.117:0/357748761 learned_addr learned my addr 172.21.15.117:0/357748761 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637a59c640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f637c0cbbd0 0x7f637c1f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f6379d9b640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f637c0cd290 0x7f637c0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 -- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f637c0cbbd0 msgr2=0x7f637c1f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f637c0cbbd0 0x7f637c1f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 -- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f637c0cd290 msgr2=0x7f637c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f637c0cd290 0x7f637c0ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 -- 172.21.15.117:0/357748761 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c0cf940 con 0x7f637c15bdc0 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.874+0000 7f637ad9d640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 0x7f637c0cf400 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f636c005d00 tx=0x7f636c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.875+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c017070 con 0x7f637c15bdc0 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.875+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f636c005330 con 0x7f637c15bdc0 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.875+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c0047a0 con 0x7f637c15bdc0 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.875+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c0c9a00 con 0x7f637c15bdc0 2024-09-17T05:27:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.875+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c0ca2c0 con 0x7f637c15bdc0 2024-09-17T05:27:44.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.876+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f637c0c9f20 con 0x7f637c15bdc0 2024-09-17T05:27:44.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.876+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f636c004940 con 0x7f637c15bdc0 2024-09-17T05:27:44.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.877+0000 7f6372ffd640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f635c078b90 0x7f635c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:44.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.877+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f636c09af50 con 0x7f637c15bdc0 2024-09-17T05:27:44.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.877+0000 7f637a59c640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f635c078b90 0x7f635c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:44.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.878+0000 7f637a59c640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f635c078b90 0x7f635c07b0a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6374035000 tx=0x7f6374038040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:44.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:44.881+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f636c0674c0 con 0x7f637c15bdc0 2024-09-17T05:27:45.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.091+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f637c0cce60 con 0x7f637c15bdc0 2024-09-17T05:27:45.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.740+0000 7f6372ffd640 1 -- 172.21.15.117:0/357748761 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v11) ==== 115+0+0 (secure 0 0 0) 0x7f636c06c370 con 0x7f637c15bdc0 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f635c078b90 msgr2=0x7f635c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f635c078b90 0x7f635c07b0a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6374035000 tx=0x7f6374038040 comp rx=0 tx=0).stop 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 msgr2=0x7f637c0cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 0x7f637c0cf400 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f636c005d00 tx=0x7f636c00dae0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 shutdown_connections 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f635c078b90 0x7f635c07b0a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f637c15bdc0 0x7f637c0cf400 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f637c0cd290 0x7f637c0ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 --2- 172.21.15.117:0/357748761 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f637c0cbbd0 0x7f637c1f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 >> 172.21.15.117:0/357748761 conn(0x7f637c06e170 msgr2=0x7f637c1513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:45.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 shutdown_connections 2024-09-17T05:27:45.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.742+0000 7f6380ccc640 1 -- 172.21.15.117:0/357748761 wait complete. 2024-09-17T05:27:45.755 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T05:27:45.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:45.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 -- 172.21.15.117:0/2436968708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 msgr2=0x7f5d68166c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:45.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/2436968708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d68166c80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005fd0 tx=0x7f5d6002a9d0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 -- 172.21.15.117:0/2436968708 shutdown_connections 2024-09-17T05:27:45.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/2436968708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d68166c80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 -- 172.21.15.117:0/2436968708 >> 172.21.15.117:0/2436968708 conn(0x7f5d68075e00 msgr2=0x7f5d6806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:45.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 -- 172.21.15.117:0/2436968708 shutdown_connections 2024-09-17T05:27:45.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.869+0000 7f5d6e65c640 1 -- 172.21.15.117:0/2436968708 wait complete. 2024-09-17T05:27:45.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.870+0000 7f5d6e65c640 1 Processor -- start 2024-09-17T05:27:45.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.871+0000 7f5d6e65c640 1 -- start start 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6e65c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d681f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6e65c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d680cb390 0x7f5d681e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6e65c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5d6815b2f0 0x7f5d68151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6e65c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5d680c5da0 con 0x7f5d680c69d0 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6e65c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6815f6b0 con 0x7f5d680cb390 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6e65c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5d681558b0 con 0x7f5d6815b2f0 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d681f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d681f54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39924/0 (socket says 172.21.15.117:39924) 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 -- 172.21.15.117:0/1055901882 learned_addr learned my addr 172.21.15.117:0/1055901882 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:45.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6cbf9640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5d6815b2f0 0x7f5d68151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d677fe640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d680cb390 0x7f5d681e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 -- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5d6815b2f0 msgr2=0x7f5d68151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5d6815b2f0 0x7f5d68151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 -- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d680cb390 msgr2=0x7f5d681e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d680cb390 0x7f5d681e6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 -- 172.21.15.117:0/1055901882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d681f1650 con 0x7f5d680c69d0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d6cbf9640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5d6815b2f0 0x7f5d68151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d67fff640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d681f54e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800c400 tx=0x7f5d5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.872+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58017070 con 0x7f5d680c69d0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.873+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d580053c0 con 0x7f5d680c69d0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.873+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58003850 con 0x7f5d680c69d0 2024-09-17T05:27:45.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.873+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d681f5a20 con 0x7f5d680c69d0 2024-09-17T05:27:45.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.873+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d681f4ce0 con 0x7f5d680c69d0 2024-09-17T05:27:45.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.873+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d680c7950 con 0x7f5d680c69d0 2024-09-17T05:27:45.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.874+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5d5801a070 con 0x7f5d680c69d0 2024-09-17T05:27:45.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.875+0000 7f5d64ff9640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5d54078bc0 0x7f5d5407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:45.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.875+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5d5809a3a0 con 0x7f5d680c69d0 2024-09-17T05:27:45.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.875+0000 7f5d677fe640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5d54078bc0 0x7f5d5407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:45.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.876+0000 7f5d677fe640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5d54078bc0 0x7f5d5407b0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d680b30e0 tx=0x7f5d5c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:45.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:45.879+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5d580625c0 con 0x7f5d680c69d0 2024-09-17T05:27:46.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.087+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d680cb810 con 0x7f5d680c69d0 2024-09-17T05:27:46.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.088+0000 7f5d64ff9640 1 -- 172.21.15.117:0/1055901882 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6898 (secure 0 0 0) 0x7f5d58061ec0 con 0x7f5d680c69d0 2024-09-17T05:27:46.088 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:46.088 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":11,"btime":"2024-09-17T05:27:45:735265+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:45.092691+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4440,"mds_1":4434},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:27:46.089 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 11 2024-09-17T05:27:46.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5d54078bc0 msgr2=0x7f5d5407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5d54078bc0 0x7f5d5407b0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d680b30e0 tx=0x7f5d5c009040 comp rx=0 tx=0).stop 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 msgr2=0x7f5d681f54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d681f54e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800c400 tx=0x7f5d5800c8c0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 shutdown_connections 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5d54078bc0 0x7f5d5407b0d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5d6815b2f0 0x7f5d68151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d680cb390 0x7f5d681e6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 --2- 172.21.15.117:0/1055901882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d680c69d0 0x7f5d681f54e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 >> 172.21.15.117:0/1055901882 conn(0x7f5d68075e00 msgr2=0x7f5d68159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:46.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 shutdown_connections 2024-09-17T05:27:46.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.090+0000 7f5d6e65c640 1 -- 172.21.15.117:0/1055901882 wait complete. 2024-09-17T05:27:46.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c93ad640 1 -- 172.21.15.117:0/3794724792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 msgr2=0x7f06b0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c93ad640 1 --2- 172.21.15.117:0/3794724792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06b0004060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00c640 tx=0x7f06b0005830 comp rx=0 tx=0).stop 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c37fe640 1 -- 172.21.15.117:0/3794724792 reap_dead start 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c93ad640 1 -- 172.21.15.117:0/3794724792 shutdown_connections 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c93ad640 1 -- 172.21.15.117:0/3794724792 >> 172.21.15.117:0/3794724792 conn(0x7f06c4074d80 msgr2=0x7f06c40d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c93ad640 1 -- 172.21.15.117:0/3794724792 shutdown_connections 2024-09-17T05:27:46.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.214+0000 7f06c93ad640 1 -- 172.21.15.117:0/3794724792 wait complete. 2024-09-17T05:27:46.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.216+0000 7f06c93ad640 1 Processor -- start 2024-09-17T05:27:46.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.216+0000 7f06c93ad640 1 -- start start 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c93ad640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06c41f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c93ad640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f06c41e1de0 0x7f06c41e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c93ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c41e2c30 0x7f06c40c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c93ad640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f06c41558b0 con 0x7f06b0003c10 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c93ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f06c40cd0d0 con 0x7f06c41e2c30 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c93ad640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f06c40cb290 con 0x7f06c41e1de0 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06c41f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06c41f1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39936/0 (socket says 172.21.15.117:39936) 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c2ffd640 1 -- 172.21.15.117:0/1694325743 learned_addr learned my addr 172.21.15.117:0/1694325743 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c27fc640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f06c41e1de0 0x7f06c41e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.217+0000 7f06c37fe640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c41e2c30 0x7f06c40c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c2ffd640 1 -- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f06c41e1de0 msgr2=0x7f06c41e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c2ffd640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f06c41e1de0 0x7f06c41e6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c2ffd640 1 -- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c41e2c30 msgr2=0x7f06c40c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c2ffd640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c41e2c30 0x7f06c40c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c2ffd640 1 -- 172.21.15.117:0/1694325743 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c4156dc0 con 0x7f06b0003c10 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c2ffd640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06c41f1ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00c640 tx=0x7f06bc004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc02b960 con 0x7f06b0003c10 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06bc02bb00 con 0x7f06b0003c10 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc0060c0 con 0x7f06b0003c10 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c41f11e0 con 0x7f06b0003c10 2024-09-17T05:27:46.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.218+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c4157340 con 0x7f06b0003c10 2024-09-17T05:27:46.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.219+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c40ce150 con 0x7f06b0003c10 2024-09-17T05:27:46.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.220+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f06bc006260 con 0x7f06b0003c10 2024-09-17T05:27:46.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.220+0000 7f06bb7fe640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f06a4078bc0 0x7f06a407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.220+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f06bc0bb5f0 con 0x7f06b0003c10 2024-09-17T05:27:46.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.221+0000 7f06c27fc640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f06a4078bc0 0x7f06a407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.221+0000 7f06c27fc640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f06a4078bc0 0x7f06a407b0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06c40c7910 tx=0x7f06ac006b10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:46.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.225+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f06c40ce150 con 0x7f06b0003c10 2024-09-17T05:27:46.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.399+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06c40cb5d0 con 0x7f06b0003c10 2024-09-17T05:27:46.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.400+0000 7f06bb7fe640 1 -- 172.21.15.117:0/1694325743 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f06bc046060 con 0x7f06b0003c10 2024-09-17T05:27:46.401 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:46.401 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f06a4078bc0 msgr2=0x7f06a407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f06a4078bc0 0x7f06a407b0d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06c40c7910 tx=0x7f06ac006b10 comp rx=0 tx=0).stop 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 msgr2=0x7f06c41f1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06c41f1ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00c640 tx=0x7f06bc004bb0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 shutdown_connections 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f06a4078bc0 0x7f06a407b0d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06c41e2c30 0x7f06c40c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f06c41e1de0 0x7f06c41e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.402+0000 7f06c93ad640 1 --2- 172.21.15.117:0/1694325743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0003c10 0x7f06c41f1ca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.403+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 >> 172.21.15.117:0/1694325743 conn(0x7f06c4074d80 msgr2=0x7f06c4154f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:46.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.403+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 shutdown_connections 2024-09-17T05:27:46.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.403+0000 7f06c93ad640 1 -- 172.21.15.117:0/1694325743 wait complete. 2024-09-17T05:27:46.414 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T05:27:46.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:27:46.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 -- 172.21.15.117:0/2551283885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef500b52b0 msgr2=0x7fef500b56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 --2- 172.21.15.117:0/2551283885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef500b52b0 0x7fef500b56c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef38009830 tx=0x7fef3802a640 comp rx=0 tx=0).stop 2024-09-17T05:27:46.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 -- 172.21.15.117:0/2551283885 shutdown_connections 2024-09-17T05:27:46.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 --2- 172.21.15.117:0/2551283885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef500b52b0 0x7fef500b56c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 -- 172.21.15.117:0/2551283885 >> 172.21.15.117:0/2551283885 conn(0x7fef50074d80 msgr2=0x7fef500d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:46.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 -- 172.21.15.117:0/2551283885 shutdown_connections 2024-09-17T05:27:46.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.526+0000 7fef5542e640 1 -- 172.21.15.117:0/2551283885 wait complete. 2024-09-17T05:27:46.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.528+0000 7fef5542e640 1 Processor -- start 2024-09-17T05:27:46.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.528+0000 7fef5542e640 1 -- start start 2024-09-17T05:27:46.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef5542e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef5006dbe0 0x7fef500c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef5542e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 0x7fef5015d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef5542e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef5015bd70 0x7fef50152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef5542e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef501558b0 con 0x7fef500b52b0 2024-09-17T05:27:46.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef5542e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef500cd0d0 con 0x7fef5015bd70 2024-09-17T05:27:46.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef5542e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fef500cb290 con 0x7fef5006dbe0 2024-09-17T05:27:46.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 0x7fef5015d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 0x7fef5015d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39952/0 (socket says 172.21.15.117:39952) 2024-09-17T05:27:46.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 -- 172.21.15.117:0/768684448 learned_addr learned my addr 172.21.15.117:0/768684448 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:46.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef4f7fe640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef5015bd70 0x7fef50152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef4effd640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef5006dbe0 0x7fef500c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 -- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef5006dbe0 msgr2=0x7fef500c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef5006dbe0 0x7fef500c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 -- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef5015bd70 msgr2=0x7fef50152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef5015bd70 0x7fef50152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.529+0000 7fef467fc640 1 -- 172.21.15.117:0/768684448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5015e550 con 0x7fef500b52b0 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.530+0000 7fef467fc640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 0x7fef5015d9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fef38009920 tx=0x7fef38033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.530+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3802ae60 con 0x7fef500b52b0 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.530+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef380333d0 con 0x7fef500b52b0 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.530+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef38005020 con 0x7fef500b52b0 2024-09-17T05:27:46.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.530+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef500c7300 con 0x7fef500b52b0 2024-09-17T05:27:46.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.530+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5015e030 con 0x7fef500b52b0 2024-09-17T05:27:46.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.531+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef500b28b0 con 0x7fef500b52b0 2024-09-17T05:27:46.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.532+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef380051e0 con 0x7fef500b52b0 2024-09-17T05:27:46.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.532+0000 7fef47fff640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef30078b90 0x7fef3007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.532+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef380bc1d0 con 0x7fef500b52b0 2024-09-17T05:27:46.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.532+0000 7fef4effd640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef30078b90 0x7fef3007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.533+0000 7fef4effd640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef30078b90 0x7fef3007b0a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fef40000f60 tx=0x7fef40009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:46.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.537+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef3803f030 con 0x7fef500b52b0 2024-09-17T05:27:46.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.712+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef500c70b0 con 0x7fef30078b90 2024-09-17T05:27:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.719+0000 7fef47fff640 1 -- 172.21.15.117:0/768684448 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332602 (secure 0 0 0) 0x7fef500c70b0 con 0x7fef30078b90 2024-09-17T05:27:46.726 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:27:46.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef30078b90 msgr2=0x7fef3007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef30078b90 0x7fef3007b0a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fef40000f60 tx=0x7fef40009040 comp rx=0 tx=0).stop 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 msgr2=0x7fef5015d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 0x7fef5015d9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fef38009920 tx=0x7fef38033df0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 shutdown_connections 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef30078b90 0x7fef3007b0a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef5015bd70 0x7fef50152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef500b52b0 0x7fef5015d9a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 --2- 172.21.15.117:0/768684448 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef5006dbe0 0x7fef500c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.722+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 >> 172.21.15.117:0/768684448 conn(0x7fef50074d80 msgr2=0x7fef50154e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.723+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 shutdown_connections 2024-09-17T05:27:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.723+0000 7fef5542e640 1 -- 172.21.15.117:0/768684448 wait complete. 2024-09-17T05:27:46.734 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-17T05:27:45.353432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218436,"kb_used_data":2208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756284,"statfs":{"total":773094113280,"available":772870434816,"internally_reserved":0,"allocated":2260992,"data_stored":2006520,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4570,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"2.1f","version":"24'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688102+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:31.688102+0000","last_peered":"2024-09-17T05:27:31.688102+0000","last_clean":"2024-09-17T05:27:31.688102+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:31.688102+0000","last_undegraded":"2024-09-17T05:27:31.688102+0000","last_fullsized":"2024-09-17T05:27:31.688102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688831+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688831+0000","last_peered":"2024-09-17T05:27:31.688831+0000","last_clean":"2024-09-17T05:27:31.688831+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688831+0000","last_undegraded":"2024-09-17T05:27:31.688831+0000","last_fullsized":"2024-09-17T05:27:31.688831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490251+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:32.490251+0000","last_peered":"2024-09-17T05:27:32.490251+0000","last_clean":"2024-09-17T05:27:32.490251+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:32.490251+0000","last_undegraded":"2024-09-17T05:27:32.490251+0000","last_fullsized":"2024-09-17T05:27:32.490251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489864+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:32.489864+0000","last_peered":"2024-09-17T05:27:32.489864+0000","last_clean":"2024-09-17T05:27:32.489864+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:32.489864+0000","last_undegraded":"2024-09-17T05:27:32.489864+0000","last_fullsized":"2024-09-17T05:27:32.489864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686433+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686433+0000","last_peered":"2024-09-17T05:27:31.686433+0000","last_clean":"2024-09-17T05:27:31.686433+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686433+0000","last_undegraded":"2024-09-17T05:27:31.686433+0000","last_fullsized":"2024-09-17T05:27:31.686433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688683+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:31.688683+0000","last_peered":"2024-09-17T05:27:31.688683+0000","last_clean":"2024-09-17T05:27:31.688683+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:31.688683+0000","last_undegraded":"2024-09-17T05:27:31.688683+0000","last_fullsized":"2024-09-17T05:27:31.688683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487972+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:32.487972+0000","last_peered":"2024-09-17T05:27:32.487972+0000","last_clean":"2024-09-17T05:27:32.487972+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:32.487972+0000","last_undegraded":"2024-09-17T05:27:32.487972+0000","last_fullsized":"2024-09-17T05:27:32.487972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490722+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:32.490722+0000","last_peered":"2024-09-17T05:27:32.490722+0000","last_clean":"2024-09-17T05:27:32.490722+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:32.490722+0000","last_undegraded":"2024-09-17T05:27:32.490722+0000","last_fullsized":"2024-09-17T05:27:32.490722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686707+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:31.686707+0000","last_peered":"2024-09-17T05:27:31.686707+0000","last_clean":"2024-09-17T05:27:31.686707+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:31.686707+0000","last_undegraded":"2024-09-17T05:27:31.686707+0000","last_fullsized":"2024-09-17T05:27:31.686707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490191+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490191+0000","last_peered":"2024-09-17T05:27:32.490191+0000","last_clean":"2024-09-17T05:27:32.490191+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490191+0000","last_undegraded":"2024-09-17T05:27:32.490191+0000","last_fullsized":"2024-09-17T05:27:32.490191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490613+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490613+0000","last_peered":"2024-09-17T05:27:32.490613+0000","last_clean":"2024-09-17T05:27:32.490613+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490613+0000","last_undegraded":"2024-09-17T05:27:32.490613+0000","last_fullsized":"2024-09-17T05:27:32.490613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488359+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:32.488359+0000","last_peered":"2024-09-17T05:27:32.488359+0000","last_clean":"2024-09-17T05:27:32.488359+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:32.488359+0000","last_undegraded":"2024-09-17T05:27:32.488359+0000","last_fullsized":"2024-09-17T05:27:32.488359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488976+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:32.488976+0000","last_peered":"2024-09-17T05:27:32.488976+0000","last_clean":"2024-09-17T05:27:32.488976+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:32.488976+0000","last_undegraded":"2024-09-17T05:27:32.488976+0000","last_fullsized":"2024-09-17T05:27:32.488976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489978+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:32.489978+0000","last_peered":"2024-09-17T05:27:32.489978+0000","last_clean":"2024-09-17T05:27:32.489978+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:32.489978+0000","last_undegraded":"2024-09-17T05:27:32.489978+0000","last_fullsized":"2024-09-17T05:27:32.489978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687718+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:31.687718+0000","last_peered":"2024-09-17T05:27:31.687718+0000","last_clean":"2024-09-17T05:27:31.687718+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:31.687718+0000","last_undegraded":"2024-09-17T05:27:31.687718+0000","last_fullsized":"2024-09-17T05:27:31.687718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489640+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489640+0000","last_peered":"2024-09-17T05:27:32.489640+0000","last_clean":"2024-09-17T05:27:32.489640+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489640+0000","last_undegraded":"2024-09-17T05:27:32.489640+0000","last_fullsized":"2024-09-17T05:27:32.489640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"24'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687876+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:31.687876+0000","last_peered":"2024-09-17T05:27:31.687876+0000","last_clean":"2024-09-17T05:27:31.687876+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:31.687876+0000","last_undegraded":"2024-09-17T05:27:31.687876+0000","last_fullsized":"2024-09-17T05:27:31.687876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488502+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:32.488502+0000","last_peered":"2024-09-17T05:27:32.488502+0000","last_clean":"2024-09-17T05:27:32.488502+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:32.488502+0000","last_undegraded":"2024-09-17T05:27:32.488502+0000","last_fullsized":"2024-09-17T05:27:32.488502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490503+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:32.490503+0000","last_peered":"2024-09-17T05:27:32.490503+0000","last_clean":"2024-09-17T05:27:32.490503+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:32.490503+0000","last_undegraded":"2024-09-17T05:27:32.490503+0000","last_fullsized":"2024-09-17T05:27:32.490503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488784+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:32.488784+0000","last_peered":"2024-09-17T05:27:32.488784+0000","last_clean":"2024-09-17T05:27:32.488784+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:32.488784+0000","last_undegraded":"2024-09-17T05:27:32.488784+0000","last_fullsized":"2024-09-17T05:27:32.488784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:40.251379+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:40.251379+0000","last_peered":"2024-09-17T05:27:40.251379+0000","last_clean":"2024-09-17T05:27:40.251379+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:40.251379+0000","last_undegraded":"2024-09-17T05:27:40.251379+0000","last_fullsized":"2024-09-17T05:27:40.251379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"24'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488820+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488820+0000","last_peered":"2024-09-17T05:27:32.488820+0000","last_clean":"2024-09-17T05:27:32.488820+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488820+0000","last_undegraded":"2024-09-17T05:27:32.488820+0000","last_fullsized":"2024-09-17T05:27:32.488820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488563+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:32.488563+0000","last_peered":"2024-09-17T05:27:32.488563+0000","last_clean":"2024-09-17T05:27:32.488563+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:32.488563+0000","last_undegraded":"2024-09-17T05:27:32.488563+0000","last_fullsized":"2024-09-17T05:27:32.488563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764209+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:31.764209+0000","last_peered":"2024-09-17T05:27:31.764209+0000","last_clean":"2024-09-17T05:27:31.764209+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:31.764209+0000","last_undegraded":"2024-09-17T05:27:31.764209+0000","last_fullsized":"2024-09-17T05:27:31.764209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488875+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488875+0000","last_peered":"2024-09-17T05:27:32.488875+0000","last_clean":"2024-09-17T05:27:32.488875+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488875+0000","last_undegraded":"2024-09-17T05:27:32.488875+0000","last_fullsized":"2024-09-17T05:27:32.488875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686092+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:31.686092+0000","last_peered":"2024-09-17T05:27:31.686092+0000","last_clean":"2024-09-17T05:27:31.686092+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:31.686092+0000","last_undegraded":"2024-09-17T05:27:31.686092+0000","last_fullsized":"2024-09-17T05:27:31.686092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"24'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490498+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:32.490498+0000","last_peered":"2024-09-17T05:27:32.490498+0000","last_clean":"2024-09-17T05:27:32.490498+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:32.490498+0000","last_undegraded":"2024-09-17T05:27:32.490498+0000","last_fullsized":"2024-09-17T05:27:32.490498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687008+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:31.687008+0000","last_peered":"2024-09-17T05:27:31.687008+0000","last_clean":"2024-09-17T05:27:31.687008+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:31.687008+0000","last_undegraded":"2024-09-17T05:27:31.687008+0000","last_fullsized":"2024-09-17T05:27:31.687008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":102645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509449,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":692501,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":102119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":102187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":102063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":106577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":331776,"data_stored":692523,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:27:46.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:27:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.886+0000 7ff4721c3640 1 -- 172.21.15.117:0/139349121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 msgr2=0x7ff46c0c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.886+0000 7ff4721c3640 1 --2- 172.21.15.117:0/139349121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 0x7ff46c0c37c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009830 tx=0x7ff45c02a510 comp rx=0 tx=0).stop 2024-09-17T05:27:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.886+0000 7ff4721c3640 1 -- 172.21.15.117:0/139349121 shutdown_connections 2024-09-17T05:27:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.886+0000 7ff4721c3640 1 --2- 172.21.15.117:0/139349121 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c15d220 0x7ff46c0c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.887+0000 7ff4721c3640 1 --2- 172.21.15.117:0/139349121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 0x7ff46c0c37c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.887+0000 7ff4721c3640 1 -- 172.21.15.117:0/139349121 >> 172.21.15.117:0/139349121 conn(0x7ff46c076430 msgr2=0x7ff46c072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.887+0000 7ff4721c3640 1 -- 172.21.15.117:0/139349121 shutdown_connections 2024-09-17T05:27:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.887+0000 7ff4721c3640 1 -- 172.21.15.117:0/139349121 wait complete. 2024-09-17T05:27:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.888+0000 7ff4721c3640 1 Processor -- start 2024-09-17T05:27:46.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.888+0000 7ff4721c3640 1 -- start start 2024-09-17T05:27:46.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 0x7ff46c0b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c153260 0x7ff46c0c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 0x7ff46c151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c0cbc30 con 0x7ff46c15d220 2024-09-17T05:27:46.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c15cd00 con 0x7ff46c0cad30 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff46c0cb230 con 0x7ff46c153260 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 0x7ff46c151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 0x7ff46c151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39966/0 (socket says 172.21.15.117:39966) 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46affd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c153260 0x7ff46c0c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 -- 172.21.15.117:0/3635202468 learned_addr learned my addr 172.21.15.117:0/3635202468 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46b7fe640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 0x7ff46c0b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 -- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c153260 msgr2=0x7ff46c0c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c153260 0x7ff46c0c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 -- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 msgr2=0x7ff46c0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 0x7ff46c0b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 -- 172.21.15.117:0/3635202468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c0c80a0 con 0x7ff46c15d220 2024-09-17T05:27:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46affd640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c153260 0x7ff46c0c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff46bfff640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 0x7ff46c151d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff464009fb0 tx=0x7ff46400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464004d90 con 0x7ff46c15d220 2024-09-17T05:27:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff46400f650 con 0x7ff46c15d220 2024-09-17T05:27:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.890+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46c1f0c50 con 0x7ff46c15d220 2024-09-17T05:27:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.891+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46400faa0 con 0x7ff46c15d220 2024-09-17T05:27:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.891+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46c0c82b0 con 0x7ff46c15d220 2024-09-17T05:27:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.891+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff46c0c4940 con 0x7ff46c15d220 2024-09-17T05:27:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.892+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff46400fc40 con 0x7ff46c15d220 2024-09-17T05:27:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.893+0000 7ff453fff640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff458078b90 0x7ff45807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.893+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff46409bcd0 con 0x7ff46c15d220 2024-09-17T05:27:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.893+0000 7ff46b7fe640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff458078b90 0x7ff45807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.894+0000 7ff46b7fe640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff458078b90 0x7ff45807b0a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff45c034000 tx=0x7ff45c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:46.897+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff464063820 con 0x7ff46c15d220 2024-09-17T05:27:47.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.069+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff46c0ca900 con 0x7ff458078b90 2024-09-17T05:27:47.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.075+0000 7ff453fff640 1 -- 172.21.15.117:0/3635202468 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332602 (secure 0 0 0) 0x7ff46c0ca900 con 0x7ff458078b90 2024-09-17T05:27:47.078 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:47.080 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.078+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff458078b90 msgr2=0x7ff45807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.078+0000 7ff4721c3640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff458078b90 0x7ff45807b0a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff45c034000 tx=0x7ff45c004730 comp rx=0 tx=0).stop 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.078+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 msgr2=0x7ff46c151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.078+0000 7ff4721c3640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 0x7ff46c151d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff464009fb0 tx=0x7ff46400c7e0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 shutdown_connections 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff458078b90 0x7ff45807b0a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46c15d220 0x7ff46c151d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff46c153260 0x7ff46c0c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 --2- 172.21.15.117:0/3635202468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff46c0cad30 0x7ff46c0b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 >> 172.21.15.117:0/3635202468 conn(0x7ff46c076430 msgr2=0x7ff46c149410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:47.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 shutdown_connections 2024-09-17T05:27:47.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.079+0000 7ff4721c3640 1 -- 172.21.15.117:0/3635202468 wait complete. 2024-09-17T05:27:47.090 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-17T05:27:45.353432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218436,"kb_used_data":2208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756284,"statfs":{"total":773094113280,"available":772870434816,"internally_reserved":0,"allocated":2260992,"data_stored":2006520,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4570,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"2.1f","version":"24'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688102+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:31.688102+0000","last_peered":"2024-09-17T05:27:31.688102+0000","last_clean":"2024-09-17T05:27:31.688102+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:31.688102+0000","last_undegraded":"2024-09-17T05:27:31.688102+0000","last_fullsized":"2024-09-17T05:27:31.688102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688831+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688831+0000","last_peered":"2024-09-17T05:27:31.688831+0000","last_clean":"2024-09-17T05:27:31.688831+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688831+0000","last_undegraded":"2024-09-17T05:27:31.688831+0000","last_fullsized":"2024-09-17T05:27:31.688831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490251+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:32.490251+0000","last_peered":"2024-09-17T05:27:32.490251+0000","last_clean":"2024-09-17T05:27:32.490251+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:32.490251+0000","last_undegraded":"2024-09-17T05:27:32.490251+0000","last_fullsized":"2024-09-17T05:27:32.490251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489864+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:32.489864+0000","last_peered":"2024-09-17T05:27:32.489864+0000","last_clean":"2024-09-17T05:27:32.489864+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:32.489864+0000","last_undegraded":"2024-09-17T05:27:32.489864+0000","last_fullsized":"2024-09-17T05:27:32.489864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686433+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686433+0000","last_peered":"2024-09-17T05:27:31.686433+0000","last_clean":"2024-09-17T05:27:31.686433+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686433+0000","last_undegraded":"2024-09-17T05:27:31.686433+0000","last_fullsized":"2024-09-17T05:27:31.686433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688683+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:31.688683+0000","last_peered":"2024-09-17T05:27:31.688683+0000","last_clean":"2024-09-17T05:27:31.688683+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:31.688683+0000","last_undegraded":"2024-09-17T05:27:31.688683+0000","last_fullsized":"2024-09-17T05:27:31.688683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487972+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:32.487972+0000","last_peered":"2024-09-17T05:27:32.487972+0000","last_clean":"2024-09-17T05:27:32.487972+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:32.487972+0000","last_undegraded":"2024-09-17T05:27:32.487972+0000","last_fullsized":"2024-09-17T05:27:32.487972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490722+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:32.490722+0000","last_peered":"2024-09-17T05:27:32.490722+0000","last_clean":"2024-09-17T05:27:32.490722+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:32.490722+0000","last_undegraded":"2024-09-17T05:27:32.490722+0000","last_fullsized":"2024-09-17T05:27:32.490722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686707+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:31.686707+0000","last_peered":"2024-09-17T05:27:31.686707+0000","last_clean":"2024-09-17T05:27:31.686707+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:31.686707+0000","last_undegraded":"2024-09-17T05:27:31.686707+0000","last_fullsized":"2024-09-17T05:27:31.686707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490191+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490191+0000","last_peered":"2024-09-17T05:27:32.490191+0000","last_clean":"2024-09-17T05:27:32.490191+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490191+0000","last_undegraded":"2024-09-17T05:27:32.490191+0000","last_fullsized":"2024-09-17T05:27:32.490191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490613+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490613+0000","last_peered":"2024-09-17T05:27:32.490613+0000","last_clean":"2024-09-17T05:27:32.490613+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490613+0000","last_undegraded":"2024-09-17T05:27:32.490613+0000","last_fullsized":"2024-09-17T05:27:32.490613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488359+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:32.488359+0000","last_peered":"2024-09-17T05:27:32.488359+0000","last_clean":"2024-09-17T05:27:32.488359+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:32.488359+0000","last_undegraded":"2024-09-17T05:27:32.488359+0000","last_fullsized":"2024-09-17T05:27:32.488359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488976+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:32.488976+0000","last_peered":"2024-09-17T05:27:32.488976+0000","last_clean":"2024-09-17T05:27:32.488976+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:32.488976+0000","last_undegraded":"2024-09-17T05:27:32.488976+0000","last_fullsized":"2024-09-17T05:27:32.488976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489978+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:32.489978+0000","last_peered":"2024-09-17T05:27:32.489978+0000","last_clean":"2024-09-17T05:27:32.489978+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:32.489978+0000","last_undegraded":"2024-09-17T05:27:32.489978+0000","last_fullsized":"2024-09-17T05:27:32.489978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687718+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:31.687718+0000","last_peered":"2024-09-17T05:27:31.687718+0000","last_clean":"2024-09-17T05:27:31.687718+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:31.687718+0000","last_undegraded":"2024-09-17T05:27:31.687718+0000","last_fullsized":"2024-09-17T05:27:31.687718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489640+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489640+0000","last_peered":"2024-09-17T05:27:32.489640+0000","last_clean":"2024-09-17T05:27:32.489640+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489640+0000","last_undegraded":"2024-09-17T05:27:32.489640+0000","last_fullsized":"2024-09-17T05:27:32.489640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"24'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687876+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:31.687876+0000","last_peered":"2024-09-17T05:27:31.687876+0000","last_clean":"2024-09-17T05:27:31.687876+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:31.687876+0000","last_undegraded":"2024-09-17T05:27:31.687876+0000","last_fullsized":"2024-09-17T05:27:31.687876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488502+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:32.488502+0000","last_peered":"2024-09-17T05:27:32.488502+0000","last_clean":"2024-09-17T05:27:32.488502+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:32.488502+0000","last_undegraded":"2024-09-17T05:27:32.488502+0000","last_fullsized":"2024-09-17T05:27:32.488502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490503+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:32.490503+0000","last_peered":"2024-09-17T05:27:32.490503+0000","last_clean":"2024-09-17T05:27:32.490503+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:32.490503+0000","last_undegraded":"2024-09-17T05:27:32.490503+0000","last_fullsized":"2024-09-17T05:27:32.490503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488784+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:32.488784+0000","last_peered":"2024-09-17T05:27:32.488784+0000","last_clean":"2024-09-17T05:27:32.488784+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:32.488784+0000","last_undegraded":"2024-09-17T05:27:32.488784+0000","last_fullsized":"2024-09-17T05:27:32.488784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:40.251379+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:40.251379+0000","last_peered":"2024-09-17T05:27:40.251379+0000","last_clean":"2024-09-17T05:27:40.251379+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:40.251379+0000","last_undegraded":"2024-09-17T05:27:40.251379+0000","last_fullsized":"2024-09-17T05:27:40.251379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"24'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488820+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488820+0000","last_peered":"2024-09-17T05:27:32.488820+0000","last_clean":"2024-09-17T05:27:32.488820+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488820+0000","last_undegraded":"2024-09-17T05:27:32.488820+0000","last_fullsized":"2024-09-17T05:27:32.488820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488563+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:32.488563+0000","last_peered":"2024-09-17T05:27:32.488563+0000","last_clean":"2024-09-17T05:27:32.488563+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:32.488563+0000","last_undegraded":"2024-09-17T05:27:32.488563+0000","last_fullsized":"2024-09-17T05:27:32.488563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764209+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:31.764209+0000","last_peered":"2024-09-17T05:27:31.764209+0000","last_clean":"2024-09-17T05:27:31.764209+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:31.764209+0000","last_undegraded":"2024-09-17T05:27:31.764209+0000","last_fullsized":"2024-09-17T05:27:31.764209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488875+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488875+0000","last_peered":"2024-09-17T05:27:32.488875+0000","last_clean":"2024-09-17T05:27:32.488875+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488875+0000","last_undegraded":"2024-09-17T05:27:32.488875+0000","last_fullsized":"2024-09-17T05:27:32.488875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686092+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:31.686092+0000","last_peered":"2024-09-17T05:27:31.686092+0000","last_clean":"2024-09-17T05:27:31.686092+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:31.686092+0000","last_undegraded":"2024-09-17T05:27:31.686092+0000","last_fullsized":"2024-09-17T05:27:31.686092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"24'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490498+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:32.490498+0000","last_peered":"2024-09-17T05:27:32.490498+0000","last_clean":"2024-09-17T05:27:32.490498+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:32.490498+0000","last_undegraded":"2024-09-17T05:27:32.490498+0000","last_fullsized":"2024-09-17T05:27:32.490498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687008+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:31.687008+0000","last_peered":"2024-09-17T05:27:31.687008+0000","last_clean":"2024-09-17T05:27:31.687008+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:31.687008+0000","last_undegraded":"2024-09-17T05:27:31.687008+0000","last_fullsized":"2024-09-17T05:27:31.687008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":102645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509449,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":692501,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":102119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":102187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":102063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":106577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":331776,"data_stored":692523,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:27:47.095 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T05:27:47.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T05:27:47.096 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T05:27:47.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 -- 172.21.15.117:0/3756733109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f740cd290 msgr2=0x7f9f740cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/3756733109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f740cd290 0x7f9f740cd6a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c005fd0 tx=0x7f9f6c02a940 comp rx=0 tx=0).stop 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 -- 172.21.15.117:0/3756733109 shutdown_connections 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/3756733109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f740cd290 0x7f9f740cd6a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/3756733109 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9f740cbbd0 0x7f9f74151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 -- 172.21.15.117:0/3756733109 >> 172.21.15.117:0/3756733109 conn(0x7f9f7406e170 msgr2=0x7f9f7406e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 -- 172.21.15.117:0/3756733109 shutdown_connections 2024-09-17T05:27:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.245+0000 7f9f7ae15640 1 -- 172.21.15.117:0/3756733109 wait complete. 2024-09-17T05:27:47.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.246+0000 7f9f7ae15640 1 Processor -- start 2024-09-17T05:27:47.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.246+0000 7f9f7ae15640 1 -- start start 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f7ae15640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f740cbbd0 0x7f9f741f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f7ae15640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9f740cd290 0x7f9f74154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f7ae15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 0x7f9f74150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f7ae15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7415f090 con 0x7f9f7415bdc0 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f7ae15640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7415edb0 con 0x7f9f740cbbd0 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f7ae15640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f9f740b43e0 con 0x7f9f740cd290 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 0x7f9f74150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 0x7f9f74150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39988/0 (socket says 172.21.15.117:39988) 2024-09-17T05:27:47.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 -- 172.21.15.117:0/2538006351 learned_addr learned my addr 172.21.15.117:0/2538006351 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f73fff640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9f740cd290 0x7f9f74154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f78bb1640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f740cbbd0 0x7f9f741f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 -- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9f740cd290 msgr2=0x7f9f74154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9f740cd290 0x7f9f74154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 -- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f740cbbd0 msgr2=0x7f9f741f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f740cbbd0 0x7f9f741f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 -- 172.21.15.117:0/2538006351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f741f2780 con 0x7f9f7415bdc0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.248+0000 7f9f793b2640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 0x7f9f74150f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009cd0 tx=0x7f9f6c031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.249+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c03e070 con 0x7f9f7415bdc0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.249+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f6c004540 con 0x7f9f7415bdc0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.249+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f741569c0 con 0x7f9f7415bdc0 2024-09-17T05:27:47.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.249+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c043460 con 0x7f9f7415bdc0 2024-09-17T05:27:47.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.249+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f740cab00 con 0x7f9f7415bdc0 2024-09-17T05:27:47.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.250+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f74050f60 con 0x7f9f7415bdc0 2024-09-17T05:27:47.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.250+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9f6c043600 con 0x7f9f7415bdc0 2024-09-17T05:27:47.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.251+0000 7f9f717fa640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9f60078bc0 0x7f9f6007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.251+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9f6c0bcb90 con 0x7f9f7415bdc0 2024-09-17T05:27:47.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.251+0000 7f9f78bb1640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9f60078bc0 0x7f9f6007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.252+0000 7f9f78bb1640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9f60078bc0 0x7f9f6007b0d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f64005fd0 tx=0x7f9f64005850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:47.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.256+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9f74050f60 con 0x7f9f7415bdc0 2024-09-17T05:27:47.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.436+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f9f740cce60 con 0x7f9f7415bdc0 2024-09-17T05:27:47.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.437+0000 7f9f717fa640 1 -- 172.21.15.117:0/2538006351 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v36) ==== 86+0+160 (secure 0 0 0) 0x7f9f6c084d20 con 0x7f9f7415bdc0 2024-09-17T05:27:47.437 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:47.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9f60078bc0 msgr2=0x7f9f6007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9f60078bc0 0x7f9f6007b0d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f64005fd0 tx=0x7f9f64005850 comp rx=0 tx=0).stop 2024-09-17T05:27:47.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 msgr2=0x7f9f74150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 0x7f9f74150f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009cd0 tx=0x7f9f6c031e20 comp rx=0 tx=0).stop 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 shutdown_connections 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9f60078bc0 0x7f9f6007b0d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f7415bdc0 0x7f9f74150f00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9f740cd290 0x7f9f74154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 --2- 172.21.15.117:0/2538006351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f740cbbd0 0x7f9f741f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 >> 172.21.15.117:0/2538006351 conn(0x7f9f7406e170 msgr2=0x7f9f740b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:47.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 shutdown_connections 2024-09-17T05:27:47.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.439+0000 7f9f7ae15640 1 -- 172.21.15.117:0/2538006351 wait complete. 2024-09-17T05:27:47.450 INFO:teuthology.orchestra.run.smithi117.stdout:[{"version":36,"timestamp":"2024-09-17T05:27:13.226651+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"4741.418695"}]}] 2024-09-17T05:27:47.450 INFO:tasks.ceph_manager:config epoch is 36 2024-09-17T05:27:47.450 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T05:27:47.451 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T05:27:47.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 -- 172.21.15.117:0/1098025718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 msgr2=0x7f7334152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 --2- 172.21.15.117:0/1098025718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f7334152210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7334152750 tx=0x7f732402a4d0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 -- 172.21.15.117:0/1098025718 shutdown_connections 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 --2- 172.21.15.117:0/1098025718 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f73340c4f50 0x7f73340c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 --2- 172.21.15.117:0/1098025718 >> v2:172.21.15.119:3300/0 conn(0x7f7334071850 0x7f7334071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 --2- 172.21.15.117:0/1098025718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f7334152210 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 -- 172.21.15.117:0/1098025718 >> 172.21.15.117:0/1098025718 conn(0x7f733406e590 msgr2=0x7f73340cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:47.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 -- 172.21.15.117:0/1098025718 shutdown_connections 2024-09-17T05:27:47.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.562+0000 7f733895f640 1 -- 172.21.15.117:0/1098025718 wait complete. 2024-09-17T05:27:47.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.564+0000 7f733895f640 1 Processor -- start 2024-09-17T05:27:47.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.564+0000 7f733895f640 1 -- start start 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.565+0000 7f733895f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7334071850 0x7f73341510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.565+0000 7f733895f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73340c4f50 0x7f73341f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.565+0000 7f733895f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f733414ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.565+0000 7f733895f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f73340ce0d0 con 0x7f73340c67a0 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.565+0000 7f733895f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f73340c58a0 con 0x7f73340c4f50 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.565+0000 7f733895f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f73340c7ff0 con 0x7f7334071850 2024-09-17T05:27:47.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f733414ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f733414ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40006/0 (socket says 172.21.15.117:40006) 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 -- 172.21.15.117:0/4115893045 learned_addr learned my addr 172.21.15.117:0/4115893045 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f733259c640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7334071850 0x7f73341510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7329d9b640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73340c4f50 0x7f73341f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 -- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7334071850 msgr2=0x7f73341510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7334071850 0x7f73341510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 -- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73340c4f50 msgr2=0x7f73341f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73340c4f50 0x7f73341f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 -- 172.21.15.117:0/4115893045 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73341f21d0 con 0x7f73340c67a0 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f7332d9d640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f733414ba90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f732c00eba0 tx=0x7f732c00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c00ec90 con 0x7f73340c67a0 2024-09-17T05:27:47.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f732c00ee30 con 0x7f73340c67a0 2024-09-17T05:27:47.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73341ee7b0 con 0x7f73340c67a0 2024-09-17T05:27:47.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c004d40 con 0x7f73340c67a0 2024-09-17T05:27:47.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.566+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73341eed70 con 0x7f73340c67a0 2024-09-17T05:27:47.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.567+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7334155f30 con 0x7f73340c67a0 2024-09-17T05:27:47.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.568+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f732c005230 con 0x7f73340c67a0 2024-09-17T05:27:47.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.568+0000 7f732b7fe640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7314078b90 0x7f731407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.569+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f732c09d560 con 0x7f73340c67a0 2024-09-17T05:27:47.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.569+0000 7f733259c640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7314078b90 0x7f731407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.569+0000 7f733259c640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7314078b90 0x7f731407b0a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7324033000 tx=0x7f7324004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:47.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.573+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7334155f30 con 0x7f73340c67a0 2024-09-17T05:27:47.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.797+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f73340c8d00 con 0x7f73340c67a0 2024-09-17T05:27:47.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.799+0000 7f732b7fe640 1 -- 172.21.15.117:0/4115893045 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+194716 (secure 0 0 0) 0x7f732c065660 con 0x7f73340c67a0 2024-09-17T05:27:47.803 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:47.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7314078b90 msgr2=0x7f731407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7314078b90 0x7f731407b0a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7324033000 tx=0x7f7324004730 comp rx=0 tx=0).stop 2024-09-17T05:27:47.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 msgr2=0x7f733414ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f733414ba90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f732c00eba0 tx=0x7f732c00b690 comp rx=0 tx=0).stop 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 shutdown_connections 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7314078b90 0x7f731407b0a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73340c67a0 0x7f733414ba90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73340c4f50 0x7f73341f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 --2- 172.21.15.117:0/4115893045 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7334071850 0x7f73341510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 >> 172.21.15.117:0/4115893045 conn(0x7f733406e590 msgr2=0x7f73341562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 shutdown_connections 2024-09-17T05:27:47.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.802+0000 7f733895f640 1 -- 172.21.15.117:0/4115893045 wait complete. 2024-09-17T05:27:47.814 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"flags":0,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":61506279},{"type":"v1","addr":"172.21.15.119:6801","nonce":61506279}]},"active_addr":"172.21.15.119:6801/61506279","active_change":"2024-09-17T05:27:05.339850+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4109,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":4199931109}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":3936929885}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2384404158}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2691383658}]}]} 2024-09-17T05:27:47.818 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T05:27:47.818 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T05:27:47.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:47.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.929+0000 7f57c58b6640 1 -- 172.21.15.117:0/2220575866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00c6800 msgr2=0x7f57c0165c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 --2- 172.21.15.117:0/2220575866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00c6800 0x7f57c0165c70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57c00763d0 tx=0x7f57b802a680 comp rx=0 tx=0).stop 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 -- 172.21.15.117:0/2220575866 shutdown_connections 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 --2- 172.21.15.117:0/2220575866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00c6800 0x7f57c0165c70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 --2- 172.21.15.117:0/2220575866 >> v2:172.21.15.119:3300/0 conn(0x7f57c015b2f0 0x7f57c00c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 -- 172.21.15.117:0/2220575866 >> 172.21.15.117:0/2220575866 conn(0x7f57c0075e00 msgr2=0x7f57c006e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 -- 172.21.15.117:0/2220575866 shutdown_connections 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.930+0000 7f57c58b6640 1 -- 172.21.15.117:0/2220575866 wait complete. 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.931+0000 7f57c58b6640 1 Processor -- start 2024-09-17T05:27:47.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.931+0000 7f57c58b6640 1 -- start start 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c00c6800 0x7f57c01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 0x7f57c00c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c015b2f0 0x7f57c00c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c00c5da0 con 0x7f57c00cb390 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57c015f6b0 con 0x7f57c00c6800 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f57c00c6c10 con 0x7f57c015b2f0 2024-09-17T05:27:47.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 0x7f57c00c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 0x7f57c00c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40022/0 (socket says 172.21.15.117:40022) 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 -- 172.21.15.117:0/855144474 learned_addr learned my addr 172.21.15.117:0/855144474 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57beffd640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c00c6800 0x7f57c01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57bf7fe640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c015b2f0 0x7f57c00c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 -- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c015b2f0 msgr2=0x7f57c00c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c015b2f0 0x7f57c00c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 -- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c00c6800 msgr2=0x7f57c01573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c00c6800 0x7f57c01573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 -- 172.21.15.117:0/855144474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c0157910 con 0x7f57c00cb390 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57be7fc640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 0x7f57c00c9970 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f57b400d6d0 tx=0x7f57b400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4017070 con 0x7f57c00cb390 2024-09-17T05:27:47.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57b4004310 con 0x7f57c00cb390 2024-09-17T05:27:47.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c01564a0 con 0x7f57c00cb390 2024-09-17T05:27:47.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.933+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4009a20 con 0x7f57c00cb390 2024-09-17T05:27:47.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.934+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c01f2560 con 0x7f57c00cb390 2024-09-17T05:27:47.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.935+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c00cab90 con 0x7f57c00cb390 2024-09-17T05:27:47.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.935+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f57b4009bc0 con 0x7f57c00cb390 2024-09-17T05:27:47.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.936+0000 7f57a7fff640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57ac078b90 0x7f57ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:47.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.936+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f57b409b080 con 0x7f57c00cb390 2024-09-17T05:27:47.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.936+0000 7f57beffd640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57ac078b90 0x7f57ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:47.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.937+0000 7f57beffd640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57ac078b90 0x7f57ac07b0a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f57c015ea60 tx=0x7f57b0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:47.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:47.940+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f57b4062b30 con 0x7f57c00cb390 2024-09-17T05:27:48.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.117+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57c00cc990 con 0x7f57c00cb390 2024-09-17T05:27:48.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.118+0000 7f57a7fff640 1 -- 172.21.15.117:0/855144474 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f57b4067560 con 0x7f57c00cb390 2024-09-17T05:27:48.119 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:48.119 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:48.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.120+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57ac078b90 msgr2=0x7f57ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57ac078b90 0x7f57ac07b0a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f57c015ea60 tx=0x7f57b0009040 comp rx=0 tx=0).stop 2024-09-17T05:27:48.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 msgr2=0x7f57c00c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 0x7f57c00c9970 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f57b400d6d0 tx=0x7f57b400db90 comp rx=0 tx=0).stop 2024-09-17T05:27:48.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 shutdown_connections 2024-09-17T05:27:48.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57ac078b90 0x7f57ac07b0a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c015b2f0 0x7f57c00c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c00cb390 0x7f57c00c9970 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 --2- 172.21.15.117:0/855144474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c00c6800 0x7f57c01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 >> 172.21.15.117:0/855144474 conn(0x7f57c0075e00 msgr2=0x7f57c00b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 shutdown_connections 2024-09-17T05:27:48.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.121+0000 7f57c58b6640 1 -- 172.21.15.117:0/855144474 wait complete. 2024-09-17T05:27:48.132 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T05:27:48.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3476669154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 msgr2=0x7f40c00b56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3476669154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00b56c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009830 tx=0x7f40a802a640 comp rx=0 tx=0).stop 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3476669154 shutdown_connections 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3476669154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00b56c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3476669154 >> 172.21.15.117:0/3476669154 conn(0x7f40c0074d80 msgr2=0x7f40c00d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3476669154 shutdown_connections 2024-09-17T05:27:48.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.252+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3476669154 wait complete. 2024-09-17T05:27:48.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.254+0000 7f40c4ea2640 1 Processor -- start 2024-09-17T05:27:48.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.254+0000 7f40c4ea2640 1 -- start start 2024-09-17T05:27:48.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40c4ea2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006dbe0 0x7f40c015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40c4ea2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40c4ea2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40c015bd70 0x7f40c0152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40c4ea2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f40c01558b0 con 0x7f40c00b52b0 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40c4ea2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f40c00cd0d0 con 0x7f40c006dbe0 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40c4ea2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f40c00cb290 con 0x7f40c015bd70 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40bdd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40bdd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40054/0 (socket says 172.21.15.117:40054) 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.255+0000 7f40bdd9b640 1 -- 172.21.15.117:0/3705182072 learned_addr learned my addr 172.21.15.117:0/3705182072 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40be59c640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006dbe0 0x7f40c015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bed9d640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40c015bd70 0x7f40c0152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bdd9b640 1 -- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40c015bd70 msgr2=0x7f40c0152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bdd9b640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40c015bd70 0x7f40c0152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bdd9b640 1 -- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006dbe0 msgr2=0x7f40c015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bdd9b640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006dbe0 0x7f40c015dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bdd9b640 1 -- 172.21.15.117:0/3705182072 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c015e100 con 0x7f40c00b52b0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40bdd9b640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00cbc30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f40a8000c00 tx=0x7f40a8033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80332e0 con 0x7f40c00b52b0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40a8043070 con 0x7f40c00b52b0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c01f14b0 con 0x7f40c00b52b0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80317b0 con 0x7f40c00b52b0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.256+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c015ecc0 con 0x7f40c00b52b0 2024-09-17T05:27:48.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.257+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c015ea30 con 0x7f40c00b52b0 2024-09-17T05:27:48.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.258+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f40a8031950 con 0x7f40c00b52b0 2024-09-17T05:27:48.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.258+0000 7f40b6ffd640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f40a0078bc0 0x7f40a007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.258+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f40a80bbf90 con 0x7f40c00b52b0 2024-09-17T05:27:48.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.258+0000 7f40be59c640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f40a0078bc0 0x7f40a007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.259+0000 7f40be59c640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f40a0078bc0 0x7f40a007b0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001ce0 tx=0x7f40b0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.263+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f40a8084090 con 0x7f40c00b52b0 2024-09-17T05:27:48.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.438+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40c0159890 con 0x7f40c00b52b0 2024-09-17T05:27:48.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.439+0000 7f40b6ffd640 1 -- 172.21.15.117:0/3705182072 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f40c0159890 con 0x7f40c00b52b0 2024-09-17T05:27:48.440 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:48.440 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:48.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f40a0078bc0 msgr2=0x7f40a007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f40a0078bc0 0x7f40a007b0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001ce0 tx=0x7f40b0009040 comp rx=0 tx=0).stop 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 msgr2=0x7f40c00cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00cbc30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f40a8000c00 tx=0x7f40a8033df0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 shutdown_connections 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f40a0078bc0 0x7f40a007b0d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40c015bd70 0x7f40c0152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40c00b52b0 0x7f40c00cbc30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 --2- 172.21.15.117:0/3705182072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006dbe0 0x7f40c015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 >> 172.21.15.117:0/3705182072 conn(0x7f40c0074d80 msgr2=0x7f40c0154d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 shutdown_connections 2024-09-17T05:27:48.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.442+0000 7f40c4ea2640 1 -- 172.21.15.117:0/3705182072 wait complete. 2024-09-17T05:27:48.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T05:27:48.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T05:27:48.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T05:27:48.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T05:27:48.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T05:27:48.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T05:27:48.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-17T05:27:48.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-17T05:27:48.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.673+0000 7f7ecbfff640 1 -- 172.21.15.117:0/1692745174 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed40afa40 con 0x7f7ecc008ed0 2024-09-17T05:27:48.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 -- 172.21.15.117:0/1692745174 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ecc008ed0 msgr2=0x7f7ecc0e62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 --2- 172.21.15.117:0/1692745174 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ecc008ed0 0x7f7ecc0e62e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40524b0 tx=0x7f7ed40a7380 comp rx=0 tx=0).stop 2024-09-17T05:27:48.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 -- 172.21.15.117:0/1692745174 shutdown_connections 2024-09-17T05:27:48.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 --2- 172.21.15.117:0/1692745174 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ecc008ed0 0x7f7ecc0e62e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7fda698b1640 1 -- 172.21.15.117:0/405914926 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034b90 msgr2=0x7fda5c0de5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7fda698b1640 1 --2- 172.21.15.117:0/405914926 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034b90 0x7fda5c0de5f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00e510 tx=0x7fda5002a400 comp rx=0 tx=0).stop 2024-09-17T05:27:48.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 --2- 172.21.15.117:0/1692745174 >> v2:172.21.15.117:3300/0 conn(0x7f7ecc0d2590 0x7f7ecc008990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f4a41554640 1 -- 172.21.15.117:0/3415675518 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b3490 msgr2=0x7f4a3c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f4a41554640 1 --2- 172.21.15.117:0/3415675518 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b3490 0x7f4a3c06d9e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009830 tx=0x7f4a2c02a5c0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 -- 172.21.15.117:0/1692745174 >> 172.21.15.117:0/1692745174 conn(0x7f7ecc050f50 msgr2=0x7f7ecc051360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 -- 172.21.15.117:0/1692745174 shutdown_connections 2024-09-17T05:27:48.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.674+0000 7f7eda307640 1 -- 172.21.15.117:0/1692745174 wait complete. 2024-09-17T05:27:48.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7f4a41554640 1 -- 172.21.15.117:0/3415675518 shutdown_connections 2024-09-17T05:27:48.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7f4a41554640 1 --2- 172.21.15.117:0/3415675518 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b3490 0x7f4a3c06d9e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7f4a41554640 1 --2- 172.21.15.117:0/3415675518 >> v2:172.21.15.119:3300/0 conn(0x7f4a3c15f0f0 0x7f4a3c0b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7f4a41554640 1 -- 172.21.15.117:0/3415675518 >> 172.21.15.117:0/3415675518 conn(0x7f4a3c074db0 msgr2=0x7f4a3c0d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7fda698b1640 1 -- 172.21.15.117:0/405914926 shutdown_connections 2024-09-17T05:27:48.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7fda698b1640 1 --2- 172.21.15.117:0/405914926 >> v2:172.21.15.119:3300/0 conn(0x7fda5c034fa0 0x7fda5c0deb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7fda698b1640 1 --2- 172.21.15.117:0/405914926 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034b90 0x7fda5c0de5f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7fda698b1640 1 --2- 172.21.15.117:0/405914926 >> v2:172.21.15.117:3300/0 conn(0x7fda5c04bbd0 0x7fda5c0d5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7fda698b1640 1 -- 172.21.15.117:0/405914926 >> 172.21.15.117:0/405914926 conn(0x7fda5c0506d0 msgr2=0x7fda5c050ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7ff1245c6640 1 -- 172.21.15.117:0/278132683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c151fa0 msgr2=0x7ff11c0c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7f4a41554640 1 -- 172.21.15.117:0/3415675518 shutdown_connections 2024-09-17T05:27:48.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7ff1245c6640 1 --2- 172.21.15.117:0/278132683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c151fa0 0x7ff11c0c5660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff10c005fd0 tx=0x7ff10c02a540 comp rx=0 tx=0).stop 2024-09-17T05:27:48.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7f4a41554640 1 -- 172.21.15.117:0/3415675518 wait complete. 2024-09-17T05:27:48.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7ff1245c6640 1 -- 172.21.15.117:0/278132683 shutdown_connections 2024-09-17T05:27:48.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7ff1245c6640 1 --2- 172.21.15.117:0/278132683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c151fa0 0x7ff11c0c5660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7ff1245c6640 1 -- 172.21.15.117:0/278132683 >> 172.21.15.117:0/278132683 conn(0x7ff11c06e590 msgr2=0x7ff11c0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7ff1245c6640 1 -- 172.21.15.117:0/278132683 shutdown_connections 2024-09-17T05:27:48.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.675+0000 7fda698b1640 1 -- 172.21.15.117:0/405914926 shutdown_connections 2024-09-17T05:27:48.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.676+0000 7ff1245c6640 1 -- 172.21.15.117:0/278132683 wait complete. 2024-09-17T05:27:48.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.676+0000 7f7518cce640 1 -- 172.21.15.117:0/2731876126 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 msgr2=0x7f751415b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.676+0000 7f7518cce640 1 --2- 172.21.15.117:0/2731876126 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 0x7f751415b000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f750c005fd0 tx=0x7f750c02a8b0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.676+0000 7f6097fff640 1 -- 172.21.15.117:0/2101802166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6090045490 msgr2=0x7f6090005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.676+0000 7f6097fff640 1 --2- 172.21.15.117:0/2101802166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6090045490 0x7f6090005c00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f609000d620 tx=0x7f608802a6d0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.676+0000 7fda698b1640 1 -- 172.21.15.117:0/405914926 wait complete. 2024-09-17T05:27:48.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f6097fff640 1 -- 172.21.15.117:0/2101802166 shutdown_connections 2024-09-17T05:27:48.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f6097fff640 1 --2- 172.21.15.117:0/2101802166 >> v2:172.21.15.117:3300/0 conn(0x7f60900e3b50 0x7f609004afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f6097fff640 1 --2- 172.21.15.117:0/2101802166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6090045490 0x7f6090005c00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7eda307640 1 Processor -- start 2024-09-17T05:27:48.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f6097fff640 1 -- 172.21.15.117:0/2101802166 >> 172.21.15.117:0/2101802166 conn(0x7f609004fe30 msgr2=0x7f6090050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7eda307640 1 -- start start 2024-09-17T05:27:48.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7518cce640 1 -- 172.21.15.117:0/2731876126 shutdown_connections 2024-09-17T05:27:48.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7518cce640 1 --2- 172.21.15.117:0/2731876126 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 0x7f751415b000 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7518cce640 1 --2- 172.21.15.117:0/2731876126 >> v2:172.21.15.119:3300/0 conn(0x7f75140c6f50 0x7f7514073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7518cce640 1 -- 172.21.15.117:0/2731876126 >> 172.21.15.117:0/2731876126 conn(0x7f75140c9a20 msgr2=0x7f75140c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f429a7fc640 1 -- 172.21.15.117:0/3066767284 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294006bb0 con 0x7f4288003a50 2024-09-17T05:27:48.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7518cce640 1 -- 172.21.15.117:0/2731876126 shutdown_connections 2024-09-17T05:27:48.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f7518cce640 1 -- 172.21.15.117:0/2731876126 wait complete. 2024-09-17T05:27:48.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f6097fff640 1 -- 172.21.15.117:0/2101802166 shutdown_connections 2024-09-17T05:27:48.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f42a1fda640 1 -- 172.21.15.117:0/3066767284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4288003a50 msgr2=0x7f4288003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f42a1fda640 1 --2- 172.21.15.117:0/3066767284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4288003a50 0x7f4288003ec0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4294006f90 tx=0x7f4288005830 comp rx=0 tx=0).stop 2024-09-17T05:27:48.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/586365215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8d4004800 msgr2=0x7fd8d4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7fd8ee8f0640 1 --2- 172.21.15.117:0/586365215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8d4004800 0x7fd8d4004c10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd8e80521d0 tx=0x7fd8d4005830 comp rx=0 tx=0).stop 2024-09-17T05:27:48.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f429bfff640 1 -- 172.21.15.117:0/3066767284 reap_dead start 2024-09-17T05:27:48.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f4a41554640 1 Processor -- start 2024-09-17T05:27:48.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f42a1fda640 1 -- 172.21.15.117:0/3066767284 shutdown_connections 2024-09-17T05:27:48.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f42a1fda640 1 -- 172.21.15.117:0/3066767284 >> 172.21.15.117:0/3066767284 conn(0x7f429c0ccbe0 msgr2=0x7f429c0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f42a1fda640 1 -- 172.21.15.117:0/3066767284 shutdown_connections 2024-09-17T05:27:48.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fd8ee0ef640 1 -- 172.21.15.117:0/586365215 reap_dead start 2024-09-17T05:27:48.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/586365215 shutdown_connections 2024-09-17T05:27:48.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/586365215 >> 172.21.15.117:0/586365215 conn(0x7fd8e004fe30 msgr2=0x7fd8e0050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.677+0000 7f6097fff640 1 -- 172.21.15.117:0/2101802166 wait complete. 2024-09-17T05:27:48.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f4a41554640 1 -- start start 2024-09-17T05:27:48.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fda698b1640 1 Processor -- start 2024-09-17T05:27:48.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/586365215 shutdown_connections 2024-09-17T05:27:48.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7ff1245c6640 1 Processor -- start 2024-09-17T05:27:48.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7ff1245c6640 1 -- start start 2024-09-17T05:27:48.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7f7eda307640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ecc008ed0 0x7f7ecc0d67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7f7eda307640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ecc04d260 0x7f7ecc0dbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7f7eda307640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ecc0d2590 0x7f7ecc0d7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7f7eda307640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc0dfaf0 con 0x7f7ecc0d2590 2024-09-17T05:27:48.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7f7eda307640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc008500 con 0x7f7ecc04d260 2024-09-17T05:27:48.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.679+0000 7f7eda307640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f7ecc0d5f20 con 0x7f7ecc008ed0 2024-09-17T05:27:48.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/586365215 wait complete. 2024-09-17T05:27:48.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ecc0d2590 0x7f7ecc0d7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ecc0d2590 0x7f7ecc0d7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40116/0 (socket says 172.21.15.117:40116) 2024-09-17T05:27:48.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 -- 172.21.15.117:0/2802724561 learned_addr learned my addr 172.21.15.117:0/2802724561 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7518cce640 1 Processor -- start 2024-09-17T05:27:48.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 -- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ecc008ed0 msgr2=0x7f7ecc0d67c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:48.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ecc008ed0 0x7f7ecc0d67c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 -- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ecc04d260 msgr2=0x7f7ecc0dbb80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:48.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ecc04d260 0x7f7ecc0dbb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7ed9b06640 1 -- 172.21.15.117:0/2802724561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc0d6300 con 0x7f7ecc0d2590 2024-09-17T05:27:48.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.680+0000 7f7518cce640 1 -- start start 2024-09-17T05:27:48.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7fda698b1640 1 -- start start 2024-09-17T05:27:48.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.678+0000 7f42a1fda640 1 -- 172.21.15.117:0/3066767284 wait complete. 2024-09-17T05:27:48.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f42a1fda640 1 Processor -- start 2024-09-17T05:27:48.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda698b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda5c034b90 0x7fda5c0dcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f6097fff640 1 Processor -- start 2024-09-17T05:27:48.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f6097fff640 1 -- start start 2024-09-17T05:27:48.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a41554640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a3c0b3490 0x7f4a3c156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a41554640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b38a0 0x7f4a3c152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda698b1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034fa0 0x7fda5c0dd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda698b1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda5c04bbd0 0x7fda5c04f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda698b1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c04c9a0 con 0x7fda5c034b90 2024-09-17T05:27:48.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda698b1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c007a40 con 0x7fda5c04bbd0 2024-09-17T05:27:48.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda698b1640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c007f60 con 0x7fda5c034fa0 2024-09-17T05:27:48.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7ff1245c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c06cda0 0x7ff11c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a41554640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a3c15f0f0 0x7f4a3c152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a41554640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c0cc550 con 0x7f4a3c0b3490 2024-09-17T05:27:48.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7ff1245c6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff11c073dd0 0x7ff11c1f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a41554640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c0b4990 con 0x7f4a3c15f0f0 2024-09-17T05:27:48.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a41554640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3c15f500 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda63fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034fa0 0x7fda5c0dd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda690b0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda5c04bbd0 0x7fda5c04f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda690b0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda5c04bbd0 0x7fda5c04f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:43914/0 (socket says 172.21.15.117:43914) 2024-09-17T05:27:48.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda690b0640 1 -- 172.21.15.117:0/1382024211 learned_addr learned my addr 172.21.15.117:0/1382024211 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b38a0 0x7f4a3c152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f7ed9b06640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ecc0d2590 0x7f7ecc0d7a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7ed406bc20 tx=0x7f7ed406be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b38a0 0x7f4a3c152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48254/0 (socket says 172.21.15.117:48254) 2024-09-17T05:27:48.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f4a3b7fe640 1 -- 172.21.15.117:0/530931922 learned_addr learned my addr 172.21.15.117:0/530931922 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a40d53640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a3c15f0f0 0x7f4a3c152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7fda688af640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda5c034b90 0x7fda5c0dcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda690b0640 1 -- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034fa0 msgr2=0x7fda5c0dd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda690b0640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034fa0 0x7fda5c0dd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda690b0640 1 -- 172.21.15.117:0/1382024211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda5c034b90 msgr2=0x7fda5c0dcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda690b0640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda5c034b90 0x7fda5c0dcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda690b0640 1 -- 172.21.15.117:0/1382024211 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c004f90 con 0x7fda5c04bbd0 2024-09-17T05:27:48.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff1245c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff11c151fa0 0x7ff11c0c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda63fff640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fda5c034fa0 0x7fda5c0dd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:48.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fd8ee8f0640 1 Processor -- start 2024-09-17T05:27:48.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff121b61640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff11c073dd0 0x7ff11c1f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff121b61640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff11c073dd0 0x7ff11c1f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48266/0 (socket says 172.21.15.117:48266) 2024-09-17T05:27:48.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff122362640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c06cda0 0x7ff11c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff121b61640 1 -- 172.21.15.117:0/2233325876 learned_addr learned my addr 172.21.15.117:0/2233325876 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff122b63640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff11c151fa0 0x7ff11c0c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c15ee90 con 0x7ff11c06cda0 2024-09-17T05:27:48.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed40b4040 con 0x7f7ecc0d2590 2024-09-17T05:27:48.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ed40bd070 con 0x7f7ecc0d2590 2024-09-17T05:27:48.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a3bfff640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a3c0b3490 0x7f4a3c156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda690b0640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda5c04bbd0 0x7fda5c04f9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fda64052860 tx=0x7fda6406a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ecc0d6d00 con 0x7f7ecc0d2590 2024-09-17T05:27:48.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c15e960 con 0x7ff11c151fa0 2024-09-17T05:27:48.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a3b7fe640 1 -- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a3c15f0f0 msgr2=0x7f4a3c152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a3b7fe640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a3c15f0f0 0x7f4a3c152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a3b7fe640 1 -- 172.21.15.117:0/530931922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a3c0b3490 msgr2=0x7f4a3c156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a3b7fe640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a3c0b3490 0x7f4a3c156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f4a3b7fe640 1 -- 172.21.15.117:0/530931922 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c1572b0 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64099070 con 0x7fda5c04bbd0 2024-09-17T05:27:48.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fd8ee8f0640 1 -- start start 2024-09-17T05:27:48.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7518cce640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75140c6f50 0x7f75140c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7518cce640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75140c8390 0x7f7514152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7518cce640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 0x7f7514154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7518cce640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7514157af0 con 0x7f75140c6f50 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7518cce640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7514157810 con 0x7f75140c8390 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7518cce640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f7514157dd0 con 0x7f751415abf0 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f751259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75140c6f50 0x7f75140c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f751259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75140c6f50 0x7f75140c30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40136/0 (socket says 172.21.15.117:40136) 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f751259c640 1 -- 172.21.15.117:0/1533100102 learned_addr learned my addr 172.21.15.117:0/1533100102 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed40b8a40 con 0x7f7ecc0d2590 2024-09-17T05:27:48.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ecc0d06e0 con 0x7f7ecc0d2590 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.681+0000 7f42a1fda640 1 -- start start 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f42a1fda640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f429c1ff000 0x7f429c0c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f42a1fda640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f429c1408a0 0x7f429c1686c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f42a1fda640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f429c168c00 0x7f429c13f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f42a1fda640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f429c0c37b0 con 0x7f429c168c00 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f42a1fda640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f429c0c3630 con 0x7f429c1408a0 2024-09-17T05:27:48.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f4a3b7fe640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b38a0 0x7f4a3c152590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009920 tx=0x7f4a2c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda640737b0 con 0x7fda5c04bbd0 2024-09-17T05:27:48.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.682+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c0368c0 con 0x7fda5c04bbd0 2024-09-17T05:27:48.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c0c3450 con 0x7fda5c04bbd0 2024-09-17T05:27:48.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f7512d9d640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 0x7f7514154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f7511d9b640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75140c8390 0x7f7514152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f429b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f429c1ff000 0x7f429c0c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f429b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f429c1ff000 0x7f429c0c6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48290/0 (socket says 172.21.15.117:48290) 2024-09-17T05:27:48.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f429b7fe640 1 -- 172.21.15.117:0/2328730575 learned_addr learned my addr 172.21.15.117:0/2328730575 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f42a1fda640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f429c15a700 con 0x7f429c1ff000 2024-09-17T05:27:48.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f751259c640 1 -- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 msgr2=0x7f7514154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f751259c640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f751415abf0 0x7f7514154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f751259c640 1 -- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75140c8390 msgr2=0x7f7514152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f751259c640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75140c8390 0x7f7514152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f751259c640 1 -- 172.21.15.117:0/1533100102 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7514153260 con 0x7f75140c6f50 2024-09-17T05:27:48.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.683+0000 7f6097fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6090045490 0x7f60900d3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f6097fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60900463f0 0x7f60900df710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429affd640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f429c1408a0 0x7f429c1686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f6097fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60900e3b50 0x7f60900d8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f6097fff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f60900e3000 con 0x7f60900e3b50 2024-09-17T05:27:48.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f6097fff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6090035f70 con 0x7f60900463f0 2024-09-17T05:27:48.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f6097fff640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f609004b4f0 con 0x7f6090045490 2024-09-17T05:27:48.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c033e10 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64073b80 con 0x7fda5c04bbd0 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429b7fe640 1 -- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f429c1408a0 msgr2=0x7f429c1686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429b7fe640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f429c1408a0 0x7f429c1686c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429b7fe640 1 -- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f429c168c00 msgr2=0x7f429c13f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429affd640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f429c1408a0 0x7f429c1686c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429bfff640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f429c168c00 0x7f429c13f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429b7fe640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f429c168c00 0x7f429c13f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429b7fe640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c169010 con 0x7f429c1ff000 2024-09-17T05:27:48.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429bfff640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f429c168c00 0x7f429c13f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:48.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f60977fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60900e3b50 0x7f60900d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f60977fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60900e3b50 0x7f60900d8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40146/0 (socket says 172.21.15.117:40146) 2024-09-17T05:27:48.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f60977fe640 1 -- 172.21.15.117:0/1114522910 learned_addr learned my addr 172.21.15.117:0/1114522910 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c15f170 con 0x7ff11c073dd0 2024-09-17T05:27:48.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c14f440 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c158740 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a2c031c50 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c005280 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f751259c640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75140c6f50 0x7f75140c30b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f750400d7a0 tx=0x7f750400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7fd8ee8f0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8e00d0520 0x7fd8e0175ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7fd8ee8f0640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8e0047c10 0x7fd8e0048020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee8f0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8e0034ad0 0x7fd8e0034ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f429b7fe640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f429c1ff000 0x7f429c0c6c40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f429402b0b0 tx=0x7f4294006c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee8f0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0045ef0 con 0x7fd8e00d0520 2024-09-17T05:27:48.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee8f0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e0045c10 con 0x7fd8e0034ad0 2024-09-17T05:27:48.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee8f0640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e00461d0 con 0x7fd8e0047c10 2024-09-17T05:27:48.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f6096ffd640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6090045490 0x7f60900d3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.684+0000 7f60967fc640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60900463f0 0x7f60900df710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504017070 con 0x7f75140c6f50 2024-09-17T05:27:48.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7504009d20 con 0x7f75140c6f50 2024-09-17T05:27:48.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7ff122362640 1 -- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff11c073dd0 msgr2=0x7ff11c1f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7ff122362640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff11c073dd0 0x7ff11c1f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7ff122362640 1 -- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff11c151fa0 msgr2=0x7ff11c0c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7ff122362640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff11c151fa0 0x7ff11c0c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7ff122362640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c0bd530 con 0x7ff11c06cda0 2024-09-17T05:27:48.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7514159c80 con 0x7f75140c6f50 2024-09-17T05:27:48.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504004220 con 0x7f75140c6f50 2024-09-17T05:27:48.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee0ef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8e0034ad0 0x7fd8e0034ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee0ef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8e0034ad0 0x7fd8e0034ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:43970/0 (socket says 172.21.15.117:43970) 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7fd8ee0ef640 1 -- 172.21.15.117:0/3110478907 learned_addr learned my addr 172.21.15.117:0/3110478907 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f60967fc640 1 -- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6090045490 msgr2=0x7f60900d3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f60967fc640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6090045490 0x7f60900d3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f60967fc640 1 -- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60900e3b50 msgr2=0x7f60900d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f60967fc640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60900e3b50 0x7f60900d8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f60967fc640 1 -- 172.21.15.117:0/1114522910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60900dfc50 con 0x7f60900463f0 2024-09-17T05:27:48.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751415a200 con 0x7f75140c6f50 2024-09-17T05:27:48.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.685+0000 7f60967fc640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60900463f0 0x7f60900df710 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6098052910 tx=0x7f609806a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8ee0ef640 1 -- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8e0047c10 msgr2=0x7fd8e0048020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8ee0ef640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8e0047c10 0x7fd8e0048020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8ee0ef640 1 -- 172.21.15.117:0/3110478907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8e00d0520 msgr2=0x7fd8e0175ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8ee0ef640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8e00d0520 0x7fd8e0175ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8ee0ef640 1 -- 172.21.15.117:0/3110478907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e0175730 con 0x7fd8e0034ad0 2024-09-17T05:27:48.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60980735b0 con 0x7f60900463f0 2024-09-17T05:27:48.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6098073840 con 0x7f60900463f0 2024-09-17T05:27:48.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098073cd0 con 0x7f60900463f0 2024-09-17T05:27:48.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a1c001030 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294031040 con 0x7f429c1ff000 2024-09-17T05:27:48.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4294006560 con 0x7f429c1ff000 2024-09-17T05:27:48.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4294003b40 con 0x7f429c1ff000 2024-09-17T05:27:48.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f75140c87a0 con 0x7f75140c6f50 2024-09-17T05:27:48.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8ee0ef640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8e0034ad0 0x7fd8e0034ee0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8011980 tx=0x7fd8d8011d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f42a1fda640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c13eed0 con 0x7f429c1ff000 2024-09-17T05:27:48.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ecc0d4050 con 0x7f7ecc0d2590 2024-09-17T05:27:48.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d800fc50 con 0x7fd8e0034ad0 2024-09-17T05:27:48.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8d8016070 con 0x7fd8e0034ad0 2024-09-17T05:27:48.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.686+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d801b6d0 con 0x7fd8e0034ad0 2024-09-17T05:27:48.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7f42a1fda640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c1fe800 con 0x7f429c1ff000 2024-09-17T05:27:48.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e00d1d70 con 0x7fd8e0034ad0 2024-09-17T05:27:48.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7ff122362640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c06cda0 0x7ff11c155c60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff11800b3e0 tx=0x7ff11800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e00d2330 con 0x7fd8e0034ad0 2024-09-17T05:27:48.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4a2c045020 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7ed40b8be0 con 0x7f7ecc0d2590 2024-09-17T05:27:48.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118015020 con 0x7ff11c06cda0 2024-09-17T05:27:48.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.687+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff118011920 con 0x7ff11c06cda0 2024-09-17T05:27:48.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f4a38ff9640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a28078c90 0x7f4a2807b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4a2c0bb770 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f4a38ff9640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f4a280823d0 0x7f4a28084860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 --> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a2c009030 con 0x7f4a280823d0 2024-09-17T05:27:48.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd8e00d5200 con 0x7fd8e0034ad0 2024-09-17T05:27:48.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4a2c0c2050 con 0x7f4a3c0b38a0 2024-09-17T05:27:48.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f42a1fda640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f429c0c8560 con 0x7f429c1ff000 2024-09-17T05:27:48.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c0bd810 con 0x7ff11c06cda0 2024-09-17T05:27:48.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f4a40d53640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f4a280823d0 0x7f4a28084860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1561f0 con 0x7ff11c06cda0 2024-09-17T05:27:48.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7ec9ffb640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7eb8078be0 0x7f7eb807b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60900de390 con 0x7f60900463f0 2024-09-17T05:27:48.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60900def10 con 0x7f60900463f0 2024-09-17T05:27:48.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118011d10 con 0x7ff11c06cda0 2024-09-17T05:27:48.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7ed9305640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7eb8078be0 0x7f7eb807b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.688+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f75040035f0 con 0x7f75140c6f50 2024-09-17T05:27:48.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7502ffd640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f74f4078b90 0x7f74f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f750409b230 con 0x7f75140c6f50 2024-09-17T05:27:48.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7502ffd640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7f74f4082250 0x7f74f40846e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 --> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7504018ec0 con 0x7f74f4082250 2024-09-17T05:27:48.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7ed413c060 con 0x7f7ecc0d2590 2024-09-17T05:27:48.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd8d801b870 con 0x7fd8e0034ad0 2024-09-17T05:27:48.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f7ec9ffb640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7f7eb8082320 0x7f7eb80847b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 --> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7eb8084e80 con 0x7f7eb8082320 2024-09-17T05:27:48.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7ed40fb8b0 con 0x7f7ecc0d2590 2024-09-17T05:27:48.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fda6409c070 con 0x7fda5c04bbd0 2024-09-17T05:27:48.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fda5c04c010 con 0x7fda5c04bbd0 2024-09-17T05:27:48.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f4a3bfff640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a28078c90 0x7f4a2807b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f60900df2f0 con 0x7f60900463f0 2024-09-17T05:27:48.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.689+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4294034070 con 0x7f429c1ff000 2024-09-17T05:27:48.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f4293fff640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f427c078c60 0x7f427c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f42940bb1d0 con 0x7f429c1ff000 2024-09-17T05:27:48.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f4293fff640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f427c082420 0x7f427c0848b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4294006270 con 0x7f427c082420 2024-09-17T05:27:48.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f42940c2050 con 0x7f429c1ff000 2024-09-17T05:27:48.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7fd8de7fc640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd8cc078be0 0x7fd8cc07b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.690+0000 7f7ed9305640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7eb8078be0 0x7f7eb807b0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0005fd0 tx=0x7f7ed0009260 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f7511d9b640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f74f4078b90 0x7f74f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f7ed8b04640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7f7eb8082320 0x7f7eb80847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff11c155750 con 0x7ff11c06cda0 2024-09-17T05:27:48.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fda617fa640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fda48078bc0 0x7fda4807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd8d809b9b0 con 0x7fd8e0034ad0 2024-09-17T05:27:48.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f7512d9d640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7f74f4082250 0x7f74f40846e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fd8ed8ee640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd8cc078be0 0x7fd8cc07b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f4a40d53640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f4a280823d0 0x7f4a28084860 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fd8de7fc640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fd8cc082320 0x7fd8cc0847b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 --> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8d8009030 con 0x7fd8cc082320 2024-09-17T05:27:48.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f75040a2050 con 0x7f75140c6f50 2024-09-17T05:27:48.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fda641175f0 con 0x7fda5c04bbd0 2024-09-17T05:27:48.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fda688af640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fda48078bc0 0x7fda4807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fda617fa640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7fda48082380 0x7fda48084810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 --> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda64098ec0 con 0x7fda48082380 2024-09-17T05:27:48.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fda6411e050 con 0x7fda5c04bbd0 2024-09-17T05:27:48.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f4a3bfff640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a28078c90 0x7f4a2807b1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a30005e50 tx=0x7f4a30005de0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd8d809de60 con 0x7fd8e0034ad0 2024-09-17T05:27:48.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f429affd640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f427c078c60 0x7f427c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7fd8ed0ed640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fd8cc082320 0x7fd8cc0847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f7511d9b640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f74f4078b90 0x7f74f407b0a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75140c9fa0 tx=0x7f7508009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.691+0000 7f429bfff640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f427c082420 0x7f427c0848b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7fda63fff640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7fda48082380 0x7fda48084810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f7512d9d640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7f74f4082250 0x7f74f40846e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6098095480 con 0x7f60900463f0 2024-09-17T05:27:48.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7fda688af640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fda48078bc0 0x7fda4807b0d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda58005fd0 tx=0x7fda5800a370 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7fd8ed0ed640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fd8cc082320 0x7fd8cc0847b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f7ed8b04640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7f7eb8082320 0x7f7eb80847b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7fd8ed8ee640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd8cc078be0 0x7fd8cc07b0f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4005eb0 tx=0x7fd8e400e040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f429affd640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f427c078c60 0x7f427c07b170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f429c0734c0 tx=0x7f428c009500 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f429bfff640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f427c082420 0x7f427c0848b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== osd.4 v2:172.21.15.119:6810/4200593796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a2c009030 con 0x7f4a280823d0 2024-09-17T05:27:48.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7fda63fff640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7fda48082380 0x7fda48084810 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff118018020 con 0x7ff11c06cda0 2024-09-17T05:27:48.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.692+0000 7f607f7fe640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6084078c10 0x7f608407b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f6096ffd640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6084078c10 0x7f608407b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6098116fe0 con 0x7f60900463f0 2024-09-17T05:27:48.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7ff112ffd640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff100078be0 0x7ff10007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== osd.7 v2:172.21.15.119:6826/2580827441 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7504018ec0 con 0x7f74f4082250 2024-09-17T05:27:48.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f607f7fe640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7f60840823d0 0x7f6084084860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff11809adf0 con 0x7ff11c06cda0 2024-09-17T05:27:48.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7ff121b61640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff100078be0 0x7ff10007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7ff112ffd640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7ff1000822a0 0x7ff100084730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:48.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f60977fe640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7f60840823d0 0x7f6084084860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 --> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6098070e50 con 0x7f60840823d0 2024-09-17T05:27:48.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== osd.0 v2:172.21.15.117:6816/2282130530 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7eb8084e80 con 0x7f7eb8082320 2024-09-17T05:27:48.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== osd.1 v2:172.21.15.117:6808/2042501374 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4294006270 con 0x7f427c082420 2024-09-17T05:27:48.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6098117310 con 0x7f60900463f0 2024-09-17T05:27:48.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.694+0000 7f60977fe640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7f60840823d0 0x7f6084084860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7ff122b63640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7ff1000822a0 0x7ff100084730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:48.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.693+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== osd.5 v2:172.21.15.119:6802/2281253283 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda64098ec0 con 0x7fda48082380 2024-09-17T05:27:48.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.694+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff11801de50 con 0x7ff1000822a0 2024-09-17T05:27:48.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.695+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff11809d410 con 0x7ff11c06cda0 2024-09-17T05:27:48.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.695+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== osd.2 v2:172.21.15.117:6800/2702715806 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6098070e50 con 0x7f60840823d0 2024-09-17T05:27:48.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.695+0000 7ff122b63640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7ff1000822a0 0x7ff100084730 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.695+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== osd.6 v2:172.21.15.119:6818/2947255794 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd8d8009030 con 0x7fd8cc082320 2024-09-17T05:27:48.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.695+0000 7ff121b61640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff100078be0 0x7ff10007b0f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009230 tx=0x7ff10c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.695+0000 7f6096ffd640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6084078c10 0x7f608407b120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6088033000 tx=0x7f6088004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:48.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.696+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== osd.3 v2:172.21.15.117:6824/1231670324 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff11801de50 con 0x7ff1000822a0 2024-09-17T05:27:48.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.722+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 --> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4a1c002dd0 con 0x7f4a280823d0 2024-09-17T05:27:48.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.722+0000 7f4a38ff9640 1 -- 172.21.15.117:0/530931922 <== osd.4 v2:172.21.15.119:6810/4200593796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4a1c002dd0 con 0x7f4a280823d0 2024-09-17T05:27:48.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.723+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f4a280823d0 msgr2=0x7f4a28084860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.723+0000 7f4a41554640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f4a280823d0 0x7f4a28084860 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.723+0000 7f42a1fda640 1 -- 172.21.15.117:0/2328730575 --> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f429c073cf0 con 0x7f427c082420 2024-09-17T05:27:48.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.723+0000 7f4293fff640 1 -- 172.21.15.117:0/2328730575 <== osd.1 v2:172.21.15.117:6808/2042501374 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f429c073cf0 con 0x7f427c082420 2024-09-17T05:27:48.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.723+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a28078c90 msgr2=0x7f4a2807b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.723+0000 7f4a41554640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a28078c90 0x7f4a2807b1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a30005e50 tx=0x7f4a30005de0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b38a0 msgr2=0x7f4a3c152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a41554640 1 --2- 172.21.15.117:0/530931922 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4a3c0b38a0 0x7f4a3c152590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009920 tx=0x7f4a2c033c20 comp rx=0 tx=0).stop 2024-09-17T05:27:48.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a40d53640 1 -- 172.21.15.117:0/530931922 reap_dead start 2024-09-17T05:27:48.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 shutdown_connections 2024-09-17T05:27:48.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 >> 172.21.15.117:0/530931922 conn(0x7f4a3c074db0 msgr2=0x7f4a3c154cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 --> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fda5c0c6ea0 con 0x7fda48082380 2024-09-17T05:27:48.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f427c082420 msgr2=0x7f427c0848b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4291ffb640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f427c082420 0x7f427c0848b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f427c078c60 msgr2=0x7f427c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4291ffb640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f427c078c60 0x7f427c07b170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f429c0734c0 tx=0x7f428c009500 comp rx=0 tx=0).stop 2024-09-17T05:27:48.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 shutdown_connections 2024-09-17T05:27:48.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f4a41554640 1 -- 172.21.15.117:0/530931922 wait complete. 2024-09-17T05:27:48.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7fda617fa640 1 -- 172.21.15.117:0/1382024211 <== osd.5 v2:172.21.15.119:6802/2281253283 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fda5c0c6ea0 con 0x7fda48082380 2024-09-17T05:27:48.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.724+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 --> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f75140c9fd0 con 0x7f74f4082250 2024-09-17T05:27:48.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f429c1ff000 msgr2=0x7f429c0c6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f4291ffb640 1 --2- 172.21.15.117:0/2328730575 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f429c1ff000 0x7f429c0c6c40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f429402b0b0 tx=0x7f4294006c20 comp rx=0 tx=0).stop 2024-09-17T05:27:48.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f429bfff640 1 -- 172.21.15.117:0/2328730575 reap_dead start 2024-09-17T05:27:48.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7502ffd640 1 -- 172.21.15.117:0/1533100102 <== osd.7 v2:172.21.15.119:6826/2580827441 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f75140c9fd0 con 0x7f74f4082250 2024-09-17T05:27:48.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7fda48082380 msgr2=0x7fda48084810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 shutdown_connections 2024-09-17T05:27:48.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 >> 172.21.15.117:0/2328730575 conn(0x7f429c0ccbe0 msgr2=0x7f429c14d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 --> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff11c15f690 con 0x7ff1000822a0 2024-09-17T05:27:48.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda698b1640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7fda48082380 0x7fda48084810 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fda48078bc0 msgr2=0x7fda4807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda698b1640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fda48078bc0 0x7fda4807b0d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda58005fd0 tx=0x7fda5800a370 comp rx=0 tx=0).stop 2024-09-17T05:27:48.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda5c04bbd0 msgr2=0x7fda5c04f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda698b1640 1 --2- 172.21.15.117:0/1382024211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda5c04bbd0 0x7fda5c04f9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fda64052860 tx=0x7fda6406a0a0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 --> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd8e0032a10 con 0x7fd8cc082320 2024-09-17T05:27:48.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7f74f4082250 msgr2=0x7f74f40846e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7518cce640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7f74f4082250 0x7f74f40846e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 --> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ecc0d3e00 con 0x7f7eb8082320 2024-09-17T05:27:48.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7fda690b0640 1 -- 172.21.15.117:0/1382024211 reap_dead start 2024-09-17T05:27:48.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7ec9ffb640 1 -- 172.21.15.117:0/2802724561 <== osd.0 v2:172.21.15.117:6816/2282130530 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7ecc0d3e00 con 0x7f7eb8082320 2024-09-17T05:27:48.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f74f4078b90 msgr2=0x7f74f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7518cce640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f74f4078b90 0x7f74f407b0a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f75140c9fa0 tx=0x7f7508009040 comp rx=0 tx=0).stop 2024-09-17T05:27:48.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7ff112ffd640 1 -- 172.21.15.117:0/2233325876 <== osd.3 v2:172.21.15.117:6824/1231670324 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff11c15f690 con 0x7ff1000822a0 2024-09-17T05:27:48.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75140c6f50 msgr2=0x7f75140c30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.725+0000 7f7518cce640 1 --2- 172.21.15.117:0/1533100102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75140c6f50 0x7f75140c30b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f750400d7a0 tx=0x7f750400dc60 comp rx=0 tx=0).stop 2024-09-17T05:27:48.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7512d9d640 1 -- 172.21.15.117:0/1533100102 reap_dead start 2024-09-17T05:27:48.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 shutdown_connections 2024-09-17T05:27:48.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f4291ffb640 1 -- 172.21.15.117:0/2328730575 wait complete. 2024-09-17T05:27:48.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 shutdown_connections 2024-09-17T05:27:48.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 >> 172.21.15.117:0/1533100102 conn(0x7f75140c9a20 msgr2=0x7f751415b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7fd8de7fc640 1 -- 172.21.15.117:0/3110478907 <== osd.6 v2:172.21.15.119:6818/2947255794 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd8e0032a10 con 0x7fd8cc082320 2024-09-17T05:27:48.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 shutdown_connections 2024-09-17T05:27:48.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7518cce640 1 -- 172.21.15.117:0/1533100102 wait complete. 2024-09-17T05:27:48.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7f7eb8082320 msgr2=0x7f7eb80847b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 shutdown_connections 2024-09-17T05:27:48.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7eda307640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7f7eb8082320 0x7f7eb80847b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 >> 172.21.15.117:0/1382024211 conn(0x7fda5c0506d0 msgr2=0x7fda5c0c13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7eb8078be0 msgr2=0x7f7eb807b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7eda307640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7eb8078be0 0x7f7eb807b0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0005fd0 tx=0x7f7ed0009260 comp rx=0 tx=0).stop 2024-09-17T05:27:48.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ecc0d2590 msgr2=0x7f7ecc0d7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7f7eda307640 1 --2- 172.21.15.117:0/2802724561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ecc0d2590 0x7f7ecc0d7a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7ed406bc20 tx=0x7f7ed406be90 comp rx=0 tx=0).stop 2024-09-17T05:27:48.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7ff1000822a0 msgr2=0x7ff100084730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7ff1245c6640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7ff1000822a0 0x7ff100084730 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 shutdown_connections 2024-09-17T05:27:48.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.726+0000 7fda698b1640 1 -- 172.21.15.117:0/1382024211 wait complete. 2024-09-17T05:27:48.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7f7ed9b06640 1 -- 172.21.15.117:0/2802724561 reap_dead start 2024-09-17T05:27:48.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 shutdown_connections 2024-09-17T05:27:48.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 >> 172.21.15.117:0/2802724561 conn(0x7f7ecc050f50 msgr2=0x7f7ecc007640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff100078be0 msgr2=0x7ff10007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff100078be0 0x7ff10007b0f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009230 tx=0x7ff10c004730 comp rx=0 tx=0).stop 2024-09-17T05:27:48.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c06cda0 msgr2=0x7ff11c155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 --2- 172.21.15.117:0/2233325876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff11c06cda0 0x7ff11c155c60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff11800b3e0 tx=0x7ff11800b8a0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fd8cc082320 msgr2=0x7fd8cc0847b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fd8cc082320 0x7fd8cc0847b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 --> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f60900de140 con 0x7f60840823d0 2024-09-17T05:27:48.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff122b63640 1 -- 172.21.15.117:0/2233325876 reap_dead start 2024-09-17T05:27:48.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 shutdown_connections 2024-09-17T05:27:48.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7f7eda307640 1 -- 172.21.15.117:0/2802724561 wait complete. 2024-09-17T05:27:48.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 shutdown_connections 2024-09-17T05:27:48.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 >> 172.21.15.117:0/2233325876 conn(0x7ff11c06e590 msgr2=0x7ff11c1591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd8cc078be0 msgr2=0x7fd8cc07b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd8cc078be0 0x7fd8cc07b0f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4005eb0 tx=0x7fd8e400e040 comp rx=0 tx=0).stop 2024-09-17T05:27:48.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8e0034ad0 msgr2=0x7fd8e0034ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 --2- 172.21.15.117:0/3110478907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8e0034ad0 0x7fd8e0034ee0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8011980 tx=0x7fd8d8011d80 comp rx=0 tx=0).stop 2024-09-17T05:27:48.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 shutdown_connections 2024-09-17T05:27:48.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7ff1245c6640 1 -- 172.21.15.117:0/2233325876 wait complete. 2024-09-17T05:27:48.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee0ef640 1 -- 172.21.15.117:0/3110478907 reap_dead start 2024-09-17T05:27:48.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 shutdown_connections 2024-09-17T05:27:48.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 >> 172.21.15.117:0/3110478907 conn(0x7fd8e004fe30 msgr2=0x7fd8e004dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 shutdown_connections 2024-09-17T05:27:48.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.727+0000 7fd8ee8f0640 1 -- 172.21.15.117:0/3110478907 wait complete. 2024-09-17T05:27:48.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.728+0000 7f607f7fe640 1 -- 172.21.15.117:0/1114522910 <== osd.2 v2:172.21.15.117:6800/2702715806 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f60900de140 con 0x7f60840823d0 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.728+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7f60840823d0 msgr2=0x7f6084084860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.728+0000 7f6097fff640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7f60840823d0 0x7f6084084860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6084078c10 msgr2=0x7f608407b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6084078c10 0x7f608407b120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6088033000 tx=0x7f6088004730 comp rx=0 tx=0).stop 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60900463f0 msgr2=0x7f60900df710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 --2- 172.21.15.117:0/1114522910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60900463f0 0x7f60900df710 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6098052910 tx=0x7f609806a0a0 comp rx=0 tx=0).stop 2024-09-17T05:27:48.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f60977fe640 1 -- 172.21.15.117:0/1114522910 reap_dead start 2024-09-17T05:27:48.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 shutdown_connections 2024-09-17T05:27:48.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 >> 172.21.15.117:0/1114522910 conn(0x7f609004fe30 msgr2=0x7f6090004970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:48.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 shutdown_connections 2024-09-17T05:27:48.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:48.729+0000 7f6097fff640 1 -- 172.21.15.117:0/1114522910 wait complete. 2024-09-17T05:27:48.782 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:48.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T05:27:48.784 INFO:teuthology.orchestra.run.smithi117.stdout:55834574858 2024-09-17T05:27:48.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T05:27:48.785 INFO:teuthology.orchestra.run.smithi117.stdout:60129542153 2024-09-17T05:27:48.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T05:27:48.786 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:48.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T05:27:48.787 INFO:teuthology.orchestra.run.smithi117.stdout:60129542153 2024-09-17T05:27:48.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T05:27:48.788 INFO:teuthology.orchestra.run.smithi117.stdout:60129542153 2024-09-17T05:27:48.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T05:27:48.789 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:48.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T05:27:48.790 INFO:teuthology.orchestra.run.smithi117.stdout:64424509450 2024-09-17T05:27:48.791 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T05:27:49.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.042+0000 7fbb7affd640 1 -- 172.21.15.117:0/421406522 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68039070 con 0x7fbb740455b0 2024-09-17T05:27:49.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.042+0000 7fbb81454640 1 -- 172.21.15.117:0/421406522 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 msgr2=0x7fbb74035790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.042+0000 7fbb81454640 1 --2- 172.21.15.117:0/421406522 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb74035790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800b9c0 tx=0x7fbb6802c1e0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 -- 172.21.15.117:0/1465373399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c06d110 msgr2=0x7f523c06d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 --2- 172.21.15.117:0/1465373399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c06d110 0x7f523c06d520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f522c009880 tx=0x7f522c02a4f0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 -- 172.21.15.117:0/1465373399 shutdown_connections 2024-09-17T05:27:49.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 --2- 172.21.15.117:0/1465373399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c06d110 0x7f523c06d520 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 -- 172.21.15.117:0/1465373399 >> 172.21.15.117:0/1465373399 conn(0x7f523c0c9a20 msgr2=0x7f523c0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f3f19ffb640 1 -- 172.21.15.117:0/159261771 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10031bf0 con 0x7f3f1c15caf0 2024-09-17T05:27:49.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fefa3a8b640 1 -- 172.21.15.117:0/4066889487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c15b220 msgr2=0x7fef9c0c4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/4066889487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c15b220 0x7fef9c0c4d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef980098b0 tx=0x7fef9802a540 comp rx=0 tx=0).stop 2024-09-17T05:27:49.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fbb81454640 1 -- 172.21.15.117:0/421406522 shutdown_connections 2024-09-17T05:27:49.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fbb81454640 1 --2- 172.21.15.117:0/421406522 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb74035790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fbb81454640 1 --2- 172.21.15.117:0/421406522 >> v2:172.21.15.117:3300/0 conn(0x7fbb740d5530 0x7fbb7404c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fbb81454640 1 -- 172.21.15.117:0/421406522 >> 172.21.15.117:0/421406522 conn(0x7fbb7404fe30 msgr2=0x7fbb74050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7ff0e149b640 1 -- 172.21.15.117:0/2245709708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc072e60 msgr2=0x7ff0dc0c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7ff0e149b640 1 --2- 172.21.15.117:0/2245709708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc072e60 0x7ff0dc0c4480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0719e0 tx=0x7ff0cc02a0c0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fefa3a8b640 1 -- 172.21.15.117:0/4066889487 shutdown_connections 2024-09-17T05:27:49.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 -- 172.21.15.117:0/1465373399 shutdown_connections 2024-09-17T05:27:49.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f524292b640 1 -- 172.21.15.117:0/1465373399 wait complete. 2024-09-17T05:27:49.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/4066889487 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c0c4420 0x7fef9c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/4066889487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c15b220 0x7fef9c0c4d20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fefa3a8b640 1 -- 172.21.15.117:0/4066889487 >> 172.21.15.117:0/4066889487 conn(0x7fef9c0ccbe0 msgr2=0x7fef9c0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f3f213da640 1 -- 172.21.15.117:0/159261771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c15caf0 msgr2=0x7f3f1c071ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7fbb81454640 1 -- 172.21.15.117:0/421406522 shutdown_connections 2024-09-17T05:27:49.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.043+0000 7f3f213da640 1 --2- 172.21.15.117:0/159261771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c15caf0 0x7f3f1c071ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009860 tx=0x7f3f1002a4d0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 -- 172.21.15.117:0/4233039504 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc0d90c0 msgr2=0x7f5efc045f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 --2- 172.21.15.117:0/4233039504 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc0d90c0 0x7f5efc045f80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f0407bf30 tx=0x7f5f040a7400 comp rx=0 tx=0).stop 2024-09-17T05:27:49.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 -- 172.21.15.117:0/4233039504 shutdown_connections 2024-09-17T05:27:49.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 --2- 172.21.15.117:0/4233039504 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc0d90c0 0x7f5efc045f80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 --2- 172.21.15.117:0/4233039504 >> v2:172.21.15.117:3300/0 conn(0x7f5efc045b70 0x7f5efc0d8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 -- 172.21.15.117:0/4233039504 >> 172.21.15.117:0/4233039504 conn(0x7f5efc0506d0 msgr2=0x7f5efc050ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7fbb81454640 1 -- 172.21.15.117:0/421406522 wait complete. 2024-09-17T05:27:49.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f3f213da640 1 -- 172.21.15.117:0/159261771 shutdown_connections 2024-09-17T05:27:49.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 -- 172.21.15.117:0/4233039504 shutdown_connections 2024-09-17T05:27:49.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f3f213da640 1 --2- 172.21.15.117:0/159261771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c15caf0 0x7f3f1c071ed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f3f213da640 1 -- 172.21.15.117:0/159261771 >> 172.21.15.117:0/159261771 conn(0x7f3f1c06d740 msgr2=0x7f3f1c072a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f3f213da640 1 -- 172.21.15.117:0/159261771 shutdown_connections 2024-09-17T05:27:49.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f3f213da640 1 -- 172.21.15.117:0/159261771 wait complete. 2024-09-17T05:27:49.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7fefa3a8b640 1 -- 172.21.15.117:0/4066889487 shutdown_connections 2024-09-17T05:27:49.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7ff0e149b640 1 -- 172.21.15.117:0/2245709708 shutdown_connections 2024-09-17T05:27:49.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7fefa3a8b640 1 -- 172.21.15.117:0/4066889487 wait complete. 2024-09-17T05:27:49.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f5f0a132640 1 -- 172.21.15.117:0/4233039504 wait complete. 2024-09-17T05:27:49.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f8bccb84640 1 -- 172.21.15.117:0/4079059091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb8004800 msgr2=0x7f8bb8004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7f8bccb84640 1 --2- 172.21.15.117:0/4079059091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb8004800 0x7f8bb8004c70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8bc000de00 tx=0x7f8bb8005830 comp rx=0 tx=0).stop 2024-09-17T05:27:49.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7ff0e149b640 1 --2- 172.21.15.117:0/2245709708 >> v2:172.21.15.117:3300/0 conn(0x7ff0dc0c85d0 0x7ff0dc074ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7ff0e149b640 1 --2- 172.21.15.117:0/2245709708 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 0x7ff0dc0c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7ff0e149b640 1 --2- 172.21.15.117:0/2245709708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc072e60 0x7ff0dc0c4480 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.044+0000 7ff0e149b640 1 -- 172.21.15.117:0/2245709708 >> 172.21.15.117:0/2245709708 conn(0x7ff0dc0ccbe0 msgr2=0x7ff0dc0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/2859863541 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 msgr2=0x7ff9a80d0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/2859863541 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 0x7ff9a80d0ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff994005fd0 tx=0x7ff99402a5d0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7f8bccb84640 1 -- 172.21.15.117:0/4079059091 shutdown_connections 2024-09-17T05:27:49.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7f8bccb84640 1 --2- 172.21.15.117:0/4079059091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bb8004800 0x7f8bb8004c70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7f8bccb84640 1 -- 172.21.15.117:0/4079059091 >> 172.21.15.117:0/4079059091 conn(0x7f8bc004fe30 msgr2=0x7f8bc0050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/2859863541 shutdown_connections 2024-09-17T05:27:49.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/2859863541 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 0x7ff9a80d0ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/2859863541 >> v2:172.21.15.119:3300/0 conn(0x7ff9a815bb90 0x7ff9a80d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/2859863541 >> 172.21.15.117:0/2859863541 conn(0x7ff9a80744d0 msgr2=0x7ff9a80748e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff0e149b640 1 -- 172.21.15.117:0/2245709708 shutdown_connections 2024-09-17T05:27:49.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.045+0000 7ff0e149b640 1 -- 172.21.15.117:0/2245709708 wait complete. 2024-09-17T05:27:49.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7fbb81454640 1 Processor -- start 2024-09-17T05:27:49.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7f5f0a132640 1 Processor -- start 2024-09-17T05:27:49.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7f524292b640 1 Processor -- start 2024-09-17T05:27:49.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7f5f0a132640 1 -- start start 2024-09-17T05:27:49.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/2859863541 shutdown_connections 2024-09-17T05:27:49.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7fbb81454640 1 -- start start 2024-09-17T05:27:49.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7f8bccb84640 1 -- 172.21.15.117:0/4079059091 shutdown_connections 2024-09-17T05:27:49.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7f8bccb84640 1 -- 172.21.15.117:0/4079059091 wait complete. 2024-09-17T05:27:49.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.047+0000 7f3f213da640 1 Processor -- start 2024-09-17T05:27:49.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.046+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/2859863541 wait complete. 2024-09-17T05:27:49.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.047+0000 7fefa3a8b640 1 Processor -- start 2024-09-17T05:27:49.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.047+0000 7fefa3a8b640 1 -- start start 2024-09-17T05:27:49.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f0a132640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc045b70 0x7f5efc0052c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f0a132640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc046d90 0x7f5efc0e1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f0a132640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 0x7f5efc0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7fbb81454640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 0x7fbb74178ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f0a132640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc0471a0 con 0x7f5efc0d90c0 2024-09-17T05:27:49.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f0a132640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc0e2f40 con 0x7f5efc045b70 2024-09-17T05:27:49.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f0a132640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5efc035f80 con 0x7f5efc046d90 2024-09-17T05:27:49.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7fbb81454640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb740cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f09931640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 0x7f5efc0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f09931640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 0x7f5efc0ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40218/0 (socket says 172.21.15.117:40218) 2024-09-17T05:27:49.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f5f09931640 1 -- 172.21.15.117:0/514066933 learned_addr learned my addr 172.21.15.117:0/514066933 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7fbb81454640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb740d5530 0x7fbb740d6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb740cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb740cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48388/0 (socket says 172.21.15.117:48388) 2024-09-17T05:27:49.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.047+0000 7f3f213da640 1 -- start start 2024-09-17T05:27:49.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7ff9ae7e6640 1 Processor -- start 2024-09-17T05:27:49.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f3f213da640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 0x7f3f1c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f3f213da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f1c0c9020 0x7f3f1c155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f3f213da640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f1c15caf0 0x7f3f1c070b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f3f213da640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c0cb8f0 con 0x7f3f1c0c9020 2024-09-17T05:27:49.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f3f213da640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c0cb770 con 0x7f3f1c15caf0 2024-09-17T05:27:49.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f3f213da640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1c152ff0 con 0x7f3f1c0c4050 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09931640 1 -- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc046d90 msgr2=0x7f5efc0e1040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09130640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc045b70 0x7f5efc0052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09931640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc046d90 0x7f5efc0e1040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09931640 1 -- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc045b70 msgr2=0x7f5efc0052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09931640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc045b70 0x7f5efc0052c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09931640 1 -- 172.21.15.117:0/514066933 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc0e09d0 con 0x7f5efc0d90c0 2024-09-17T05:27:49.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7ff9ae7e6640 1 -- start start 2024-09-17T05:27:49.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f8bccb84640 1 Processor -- start 2024-09-17T05:27:49.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7ff0e149b640 1 Processor -- start 2024-09-17T05:27:49.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7ff0e149b640 1 -- start start 2024-09-17T05:27:49.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7fefa3a8b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c074cd0 0x7fef9c1579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f8bccb84640 1 -- start start 2024-09-17T05:27:49.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa3a8b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c0c4420 0x7fef9c1562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7f5f09931640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 0x7f5efc0ccfc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f0406bc20 tx=0x7f5f040ad5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7fbb7bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 0x7fbb74178ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.049+0000 7fbb7bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 0x7fbb74178ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44024/0 (socket says 172.21.15.117:44024) 2024-09-17T05:27:49.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fbb81454640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbb740459c0 con 0x7fbb740d5530 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa1827640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c074cd0 0x7fef9c1579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa1827640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c074cd0 0x7fef9c1579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40234/0 (socket says 172.21.15.117:40234) 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa3a8b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c15b220 0x7fef9c14f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa1827640 1 -- 172.21.15.117:0/3961183602 learned_addr learned my addr 172.21.15.117:0/3961183602 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa1026640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c0c4420 0x7fef9c1562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c0744d0 con 0x7fef9c074cd0 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c0741f0 con 0x7fef9c0c4420 2024-09-17T05:27:49.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fef9c0c4830 con 0x7fef9c15b220 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040bc070 con 0x7f5efc0d90c0 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f040adc40 con 0x7f5efc0d90c0 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f040bc070 con 0x7f5efc0d90c0 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5efc005800 con 0x7f5efc0d90c0 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.048+0000 7f524292b640 1 -- start start 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f524292b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c06d110 0x7f523c1f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f524292b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f523c0c66f0 0x7f523c0c3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fefa2028640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c15b220 0x7fef9c14f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fefa1026640 1 -- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c15b220 msgr2=0x7fef9c14f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fefa1026640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c15b220 0x7fef9c14f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fefa1026640 1 -- 172.21.15.117:0/3961183602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c074cd0 msgr2=0x7fef9c1579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fefa1026640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c074cd0 0x7fef9c1579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fefa1026640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9c156820 con 0x7fef9c0c4420 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5efc0e0070 con 0x7f5efc0d90c0 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f3f1affd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 0x7f3f1c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f3f1affd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 0x7f3f1c1547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48422/0 (socket says 172.21.15.117:48422) 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f3f1affd640 1 -- 172.21.15.117:0/2220930334 learned_addr learned my addr 172.21.15.117:0/2220930334 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff0e149b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0dc072e60 0x7ff0dc15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff0e149b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 0x7ff0dc15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff0e149b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc0c85d0 0x7ff0dc1627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff0e149b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc08cd10 con 0x7ff0dc072e60 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff0e149b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc16bb80 con 0x7ff0dc0c85d0 2024-09-17T05:27:49.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff0e149b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff0dc0750e0 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f524292b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c0c7ad0 0x7f523c1574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f524292b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f523c0b3170 con 0x7f523c0c7ad0 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f524292b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f523c06da60 con 0x7f523c06d110 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f524292b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f523c073f30 con 0x7f523c0c66f0 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c06d110 0x7f523c1f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c06d110 0x7f523c1f6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44066/0 (socket says 172.21.15.117:44066) 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523bfff640 1 -- 172.21.15.117:0/1666909744 learned_addr learned my addr 172.21.15.117:0/1666909744 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523b7fe640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f523c0c66f0 0x7f523c0c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f5240ec8640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c0c7ad0 0x7f523c1574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523b7fe640 1 -- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c06d110 msgr2=0x7f523c1f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523b7fe640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c06d110 0x7f523c1f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523b7fe640 1 -- 172.21.15.117:0/1666909744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c0c7ad0 msgr2=0x7f523c1574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523b7fe640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c0c7ad0 0x7f523c1574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7f523b7fe640 1 -- 172.21.15.117:0/1666909744 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523c1f6fb0 con 0x7f523c0c66f0 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f523b7fe640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f523c0c66f0 0x7f523c0c3360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f523c0760b0 tx=0x7f522c0045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.050+0000 7fbb7b7fe640 1 -- 172.21.15.117:0/3195189473 learned_addr learned my addr 172.21.15.117:0/3195189473 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fbb80c53640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb740d5530 0x7fbb740d6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74035040 con 0x7fbb74035cd0 2024-09-17T05:27:49.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fbb74035320 con 0x7fbb740455b0 2024-09-17T05:27:49.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7fefa1026640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c0c4420 0x7fef9c1562e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fef9400cc50 tx=0x7fef9400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f3f1affd640 1 -- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f1c15caf0 msgr2=0x7f3f1c070b90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:49.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f3f1affd640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f1c15caf0 0x7f3f1c070b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c062070 con 0x7f523c0c66f0 2024-09-17T05:27:49.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7ff9ae7e6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9a80cb280 0x7ff9a8070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f3f1affd640 1 -- 172.21.15.117:0/2220930334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f1c0c9020 msgr2=0x7f3f1c155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f3f1affd640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f1c0c9020 0x7f3f1c155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f3f1affd640 1 -- 172.21.15.117:0/2220930334 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1c155850 con 0x7f3f1c0c4050 2024-09-17T05:27:49.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f3f1affd640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 0x7f3f1c1547a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00b420 tx=0x7f3f0c00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f522c0315b0 con 0x7f523c0c66f0 2024-09-17T05:27:49.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c031a00 con 0x7f523c0c66f0 2024-09-17T05:27:49.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523c1f9a90 con 0x7f523c0c66f0 2024-09-17T05:27:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb7bfff640 1 -- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 msgr2=0x7fbb740cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb7bfff640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb740cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb7bfff640 1 -- 172.21.15.117:0/3195189473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb740d5530 msgr2=0x7fbb740d6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7ff9ae7e6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 0x7ff9a8071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb7bfff640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb740d5530 0x7fbb740d6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb7bfff640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb74004a10 con 0x7fbb74035cd0 2024-09-17T05:27:49.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.051+0000 7f8bccb84640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bb8004800 0x7f8bc016aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bccb84640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc0175a30 0x7f8bc0049840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bccb84640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8bc00d7c80 0x7f8bc00d8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bccb84640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc00d5200 con 0x7f8bc0175a30 2024-09-17T05:27:49.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bccb84640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc00461d0 con 0x7f8bb8004800 2024-09-17T05:27:49.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bccb84640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc00db6c0 con 0x7f8bc00d7c80 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bc7fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8bc00d7c80 0x7f8bc00d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bc77fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bb8004800 0x7f8bc016aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bc6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc0175a30 0x7f8bc0049840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb7bfff640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 0x7fbb74178ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000b980 tx=0x7fbb7000be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bc7fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8bc00d7c80 0x7f8bc00d8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48426/0 (socket says 172.21.15.117:48426) 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bc6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc0175a30 0x7f8bc0049840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40252/0 (socket says 172.21.15.117:40252) 2024-09-17T05:27:49.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f8bc7fff640 1 -- 172.21.15.117:0/946650301 learned_addr learned my addr 172.21.15.117:0/946650301 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7ff0daffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0dc072e60 0x7ff0dc15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5efc0de5f0 con 0x7f5efc0d90c0 2024-09-17T05:27:49.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70008b00 con 0x7fbb74035cd0 2024-09-17T05:27:49.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7ff0daffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0dc072e60 0x7ff0dc15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40256/0 (socket says 172.21.15.117:40256) 2024-09-17T05:27:49.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7ff0daffd640 1 -- 172.21.15.117:0/1367031006 learned_addr learned my addr 172.21.15.117:0/1367031006 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7ff0da7fc640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 0x7ff0dc15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7ff0db7fe640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc0c85d0 0x7ff0dc1627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb70008ca0 con 0x7fbb74035cd0 2024-09-17T05:27:49.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb74004cf0 con 0x7fbb74035cd0 2024-09-17T05:27:49.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb741789d0 con 0x7fbb74035cd0 2024-09-17T05:27:49.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7001e650 con 0x7fbb74035cd0 2024-09-17T05:27:49.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f523c1f5c60 con 0x7f523c0c66f0 2024-09-17T05:27:49.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f8bc77fe640 1 -- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8bc00d7c80 msgr2=0x7f8bc00d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f8bc77fe640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8bc00d7c80 0x7f8bc00d8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f8bc77fe640 1 -- 172.21.15.117:0/946650301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc0175a30 msgr2=0x7f8bc0049840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f8bc77fe640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc0175a30 0x7f8bc0049840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f8bc77fe640 1 -- 172.21.15.117:0/946650301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc00d85d0 con 0x7f8bb8004800 2024-09-17T05:27:49.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94016020 con 0x7fef9c0c4420 2024-09-17T05:27:49.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef94012420 con 0x7fef9c0c4420 2024-09-17T05:27:49.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.052+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94012850 con 0x7fef9c0c4420 2024-09-17T05:27:49.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9c1f5e00 con 0x7fef9c0c4420 2024-09-17T05:27:49.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f8bc77fe640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bb8004800 0x7f8bc016aa20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc004c50 tx=0x7f8bbc001800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9c1f63c0 con 0x7fef9c0c4420 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9ae7e6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9a815bb90 0x7ff9a8151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9ae7e6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a80c7860 con 0x7ff9a815bb90 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9ae7e6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a8155570 con 0x7ff9a80cb280 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9ae7e6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a80c79e0 con 0x7ff9a80cc740 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9a7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9a80cb280 0x7ff9a8070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9a7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9a80cb280 0x7ff9a8070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44086/0 (socket says 172.21.15.117:44086) 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9a7fff640 1 -- 172.21.15.117:0/1616323137 learned_addr learned my addr 172.21.15.117:0/1616323137 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:49.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7ff9acd83640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9a815bb90 0x7ff9a8151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f523c0c6b00 con 0x7f523c0c66f0 2024-09-17T05:27:49.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff0da7fc640 1 -- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc0c85d0 msgr2=0x7ff0dc1627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9a77fe640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 0x7ff9a8071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff0da7fc640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc0c85d0 0x7ff0dc1627c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff0da7fc640 1 -- 172.21.15.117:0/1367031006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0dc072e60 msgr2=0x7ff0dc15cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff0da7fc640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0dc072e60 0x7ff0dc15cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff0da7fc640 1 -- 172.21.15.117:0/1367031006 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc15d140 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.053+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c011830 con 0x7f3f1c0c4050 2024-09-17T05:27:49.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f0c016070 con 0x7f3f1c0c4050 2024-09-17T05:27:49.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c011d40 con 0x7f3f1c0c4050 2024-09-17T05:27:49.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f3f213da640 1 -- 172.21.15.117:0/2220930334 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1c0c9f60 con 0x7f3f1c0c4050 2024-09-17T05:27:49.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9acd83640 1 -- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 msgr2=0x7ff9a8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9acd83640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 0x7ff9a8071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9acd83640 1 -- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9a80cb280 msgr2=0x7ff9a8070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9acd83640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9a80cb280 0x7ff9a8070680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9acd83640 1 -- 172.21.15.117:0/1616323137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a8070bc0 con 0x7ff9a815bb90 2024-09-17T05:27:49.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc015020 con 0x7f8bb8004800 2024-09-17T05:27:49.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f3f213da640 1 -- 172.21.15.117:0/2220930334 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f1c14eda0 con 0x7f3f1c0c4050 2024-09-17T05:27:49.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7ff9acd83640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9a815bb90 0x7ff9a8151cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff994031000 tx=0x7ff994003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bbc011870 con 0x7f8bb8004800 2024-09-17T05:27:49.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.054+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc011ca0 con 0x7f8bb8004800 2024-09-17T05:27:49.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.055+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc00d38a0 con 0x7f8bb8004800 2024-09-17T05:27:49.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.055+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc0175230 con 0x7f8bb8004800 2024-09-17T05:27:49.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.055+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f522c065070 con 0x7f523c0c66f0 2024-09-17T05:27:49.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.055+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5c005290 con 0x7fbb74035cd0 2024-09-17T05:27:49.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.055+0000 7f5238ff9640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5228078bc0 0x7f522807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef94019020 con 0x7fef9c0c4420 2024-09-17T05:27:49.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f522c0de140 con 0x7f523c0c66f0 2024-09-17T05:27:49.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7f3f213da640 1 -- 172.21.15.117:0/2220930334 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3efc005290 con 0x7f3f1c0c4050 2024-09-17T05:27:49.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5f040ae670 con 0x7f5efc0d90c0 2024-09-17T05:27:49.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbb7001e7f0 con 0x7fbb74035cd0 2024-09-17T05:27:49.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7ff0db7fe640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc0c85d0 0x7ff0dc1627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:49.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7fbb78ff9640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbb60078c60 0x7fbb6007b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.056+0000 7fef927fc640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef80078c10 0x7fef8007b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef9409b080 con 0x7fef9c0c4420 2024-09-17T05:27:49.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbb700a3eb0 con 0x7fbb74035cd0 2024-09-17T05:27:49.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7f523bfff640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5228078bc0 0x7f522807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7ff0da7fc640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 0x7ff0dc15b4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009650 tx=0x7ff0d0009b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7fbb7b7fe640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbb60078c60 0x7fbb6007b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7f5ef5ffb640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5ee8078f90 0x7f5ee807b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f0c01b460 con 0x7f3f1c0c4050 2024-09-17T05:27:49.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5f040ad950 con 0x7f5efc0d90c0 2024-09-17T05:27:49.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc0032a10 con 0x7f8bb8004800 2024-09-17T05:27:49.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7fefa1827640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef80078c10 0x7fef8007b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7f5f09130640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5ee8078f90 0x7f5ee807b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.057+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9c1f5b90 con 0x7fef9c0c4420 2024-09-17T05:27:49.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f3f037fe640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f08078c60 0x7f3f0807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8bbc018020 con 0x7f8bb8004800 2024-09-17T05:27:49.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3f0c09abd0 con 0x7f3f1c0c4050 2024-09-17T05:27:49.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f3f1a7fc640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f08078c60 0x7f3f0807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f523bfff640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5228078bc0 0x7f522807b0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5230009990 tx=0x7f5230009110 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7fbb7b7fe640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbb60078c60 0x7fbb6007b170 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbb740df770 tx=0x7fbb6802e040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7fefa1827640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef80078c10 0x7fef8007b120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef98000c00 tx=0x7fef98004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f5f09130640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5ee8078f90 0x7f5ee807b4a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800a750 tx=0x7f5ef8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f8bb3fff640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8bac078b90 0x7f8bac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8bbc09b980 con 0x7f8bb8004800 2024-09-17T05:27:49.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f3f1a7fc640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f08078c60 0x7f3f0807b170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c06f470 tx=0x7f3f14009250 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.058+0000 7f8bc6ffd640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8bac078b90 0x7f8bac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.059+0000 7f8bc6ffd640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8bac078b90 0x7f8bac07b0a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4033000 tx=0x7f8bc0045b00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.060+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99403e030 con 0x7ff9a815bb90 2024-09-17T05:27:49.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.060+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5efc0de5f0 con 0x7f5efc0d90c0 2024-09-17T05:27:49.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff994003ba0 con 0x7ff9a815bb90 2024-09-17T05:27:49.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a815e270 con 0x7ff9a815bb90 2024-09-17T05:27:49.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994035040 con 0x7ff9a815bb90 2024-09-17T05:27:49.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a80c99e0 con 0x7ff9a815bb90 2024-09-17T05:27:49.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0004020 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0dc2071b0 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0dc15c400 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.061+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f522c0a5be0 con 0x7f523c0c66f0 2024-09-17T05:27:49.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.062+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a80cbe10 con 0x7ff9a815bb90 2024-09-17T05:27:49.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.062+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0dc15ed40 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.062+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbb7006b9d0 con 0x7fbb74035cd0 2024-09-17T05:27:49.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.063+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff994041030 con 0x7ff9a815bb90 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.064+0000 7ff9a4ff9640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff98c078bc0 0x7ff98c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.064+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff9940bb2a0 con 0x7ff9a815bb90 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.065+0000 7ff9a7fff640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff98c078bc0 0x7ff98c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.065+0000 7ff9a7fff640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff98c078bc0 0x7ff98c07b0d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009b80 tx=0x7ff99c005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.066+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3f0c020020 con 0x7f3f1c0c4050 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.066+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0d00041c0 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.066+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d000c850 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.067+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef94020070 con 0x7fef9c0c4420 2024-09-17T05:27:49.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.069+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff0d000c9f0 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.069+0000 7ff0c37fe640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff0c8078bc0 0x7ff0c807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:49.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.070+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff0d009b420 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.070+0000 7ff0daffd640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff0c8078bc0 0x7ff0c807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:49.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.070+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff994082df0 con 0x7ff9a815bb90 2024-09-17T05:27:49.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.071+0000 7ff0daffd640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff0c8078bc0 0x7ff0c807b0d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc000c00 tx=0x7ff0cc004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:49.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.072+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff0d0024070 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.072+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8bbc01f020 con 0x7f8bb8004800 2024-09-17T05:27:49.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.480+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbb5c005580 con 0x7fbb74035cd0 2024-09-17T05:27:49.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.481+0000 7fbb78ff9640 1 -- 172.21.15.117:0/3195189473 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb70070400 con 0x7fbb74035cd0 2024-09-17T05:27:49.481 INFO:teuthology.orchestra.run.smithi117.stdout:60129542152 2024-09-17T05:27:49.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.482+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5efc0e22a0 con 0x7f5efc0d90c0 2024-09-17T05:27:49.484 INFO:teuthology.orchestra.run.smithi117.stdout:55834574856 2024-09-17T05:27:49.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.483+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f523c1e7ee0 con 0x7f523c0c66f0 2024-09-17T05:27:49.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.483+0000 7f5238ff9640 1 -- 172.21.15.117:0/1666909744 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f522c0aa610 con 0x7f523c0c66f0 2024-09-17T05:27:49.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7f5ef5ffb640 1 -- 172.21.15.117:0/514066933 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5f040d0080 con 0x7f5efc0d90c0 2024-09-17T05:27:49.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7f3f213da640 1 -- 172.21.15.117:0/2220930334 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3efc005580 con 0x7f3f1c0c4050 2024-09-17T05:27:49.485 INFO:teuthology.orchestra.run.smithi117.stdout:64424509448 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbb60078c60 msgr2=0x7fbb6007b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbb60078c60 0x7fbb6007b170 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbb740df770 tx=0x7fbb6802e040 comp rx=0 tx=0).stop 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 msgr2=0x7fbb74178ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 0x7fbb74178ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000b980 tx=0x7fbb7000be40 comp rx=0 tx=0).stop 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 shutdown_connections 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbb60078c60 0x7fbb6007b170 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb740d5530 0x7fbb740d6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbb740455b0 0x7fbb740cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 --2- 172.21.15.117:0/3195189473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb74035cd0 0x7fbb74178ef0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.484+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 >> 172.21.15.117:0/3195189473 conn(0x7fbb7404fe30 msgr2=0x7fbb740de6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.485+0000 7f3f037fe640 1 -- 172.21.15.117:0/2220930334 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3f0c062d40 con 0x7f3f1c0c4050 2024-09-17T05:27:49.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.485+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 shutdown_connections 2024-09-17T05:27:49.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.485+0000 7fbb81454640 1 -- 172.21.15.117:0/3195189473 wait complete. 2024-09-17T05:27:49.488 INFO:teuthology.orchestra.run.smithi117.stdout:64424509448 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5228078bc0 msgr2=0x7f522807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5228078bc0 0x7f522807b0d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5230009990 tx=0x7f5230009110 comp rx=0 tx=0).stop 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f523c0c66f0 msgr2=0x7f523c0c3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f523c0c66f0 0x7f523c0c3360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f523c0760b0 tx=0x7f522c0045a0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 shutdown_connections 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5228078bc0 0x7f522807b0d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f523c0c7ad0 0x7f523c1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f523c0c66f0 0x7f523c0c3360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 --2- 172.21.15.117:0/1666909744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c06d110 0x7f523c1f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 >> 172.21.15.117:0/1666909744 conn(0x7f523c0c9a20 msgr2=0x7f523c1594c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.487+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 shutdown_connections 2024-09-17T05:27:49.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f524292b640 1 -- 172.21.15.117:0/1666909744 wait complete. 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 -- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f08078c60 msgr2=0x7f3f0807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f08078c60 0x7f3f0807b170 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c06f470 tx=0x7f3f14009250 comp rx=0 tx=0).stop 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 -- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 msgr2=0x7f3f1c1547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 0x7f3f1c1547a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00b420 tx=0x7f3f0c00b8e0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 -- 172.21.15.117:0/2220930334 shutdown_connections 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f08078c60 0x7f3f0807b170 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f1c15caf0 0x7f3f1c070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f1c0c9020 0x7f3f1c155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 --2- 172.21.15.117:0/2220930334 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3f1c0c4050 0x7f3f1c1547a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 -- 172.21.15.117:0/2220930334 >> 172.21.15.117:0/2220930334 conn(0x7f3f1c06d740 msgr2=0x7f3f1c072a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 -- 172.21.15.117:0/2220930334 shutdown_connections 2024-09-17T05:27:49.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.488+0000 7f3f017fa640 1 -- 172.21.15.117:0/2220930334 wait complete. 2024-09-17T05:27:49.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.489+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5ee8078f90 msgr2=0x7f5ee807b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.489+0000 7f5f0a132640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5ee8078f90 0x7f5ee807b4a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800a750 tx=0x7f5ef8008040 comp rx=0 tx=0).stop 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.489+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 msgr2=0x7f5efc0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.489+0000 7f5f0a132640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 0x7f5efc0ccfc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5f0406bc20 tx=0x7f5f040ad5d0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 shutdown_connections 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5ee8078f90 0x7f5ee807b4a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5efc0d90c0 0x7f5efc0ccfc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5efc046d90 0x7f5efc0e1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 --2- 172.21.15.117:0/514066933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5efc045b70 0x7f5efc0052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 >> 172.21.15.117:0/514066933 conn(0x7f5efc0506d0 msgr2=0x7f5efc04c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 shutdown_connections 2024-09-17T05:27:49.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.490+0000 7f5f0a132640 1 -- 172.21.15.117:0/514066933 wait complete. 2024-09-17T05:27:49.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.492+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fef9c0733b0 con 0x7fef9c0c4420 2024-09-17T05:27:49.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.493+0000 7fef927fc640 1 -- 172.21.15.117:0/3961183602 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fef940632f0 con 0x7fef9c0c4420 2024-09-17T05:27:49.496 INFO:teuthology.orchestra.run.smithi117.stdout:60129542152 2024-09-17T05:27:49.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.496+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8bc004bf40 con 0x7f8bb8004800 2024-09-17T05:27:49.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.497+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff9a8074c10 con 0x7ff9a815bb90 2024-09-17T05:27:49.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.497+0000 7f8bb3fff640 1 -- 172.21.15.117:0/946650301 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8bc00d85d0 con 0x7f8bb8004800 2024-09-17T05:27:49.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.497+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef80078c10 msgr2=0x7fef8007b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.497+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef80078c10 0x7fef8007b120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef98000c00 tx=0x7fef98004730 comp rx=0 tx=0).stop 2024-09-17T05:27:49.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.497+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c0c4420 msgr2=0x7fef9c1562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.497+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c0c4420 0x7fef9c1562e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fef9400cc50 tx=0x7fef9400a430 comp rx=0 tx=0).stop 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 shutdown_connections 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef80078c10 0x7fef8007b120 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c15b220 0x7fef9c14f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c0c4420 0x7fef9c1562e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 --2- 172.21.15.117:0/3961183602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c074cd0 0x7fef9c1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 >> 172.21.15.117:0/3961183602 conn(0x7fef9c0ccbe0 msgr2=0x7fef9c158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 shutdown_connections 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.498+0000 7fefa3a8b640 1 -- 172.21.15.117:0/3961183602 wait complete. 2024-09-17T05:27:49.499 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:49.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.499+0000 7ff9a4ff9640 1 -- 172.21.15.117:0/1616323137 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff994087820 con 0x7ff9a815bb90 2024-09-17T05:27:49.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.499+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff0dc0c7360 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.500 INFO:teuthology.orchestra.run.smithi117.stdout:60129542152 2024-09-17T05:27:49.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.500+0000 7ff0c37fe640 1 -- 172.21.15.117:0/1367031006 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff0d0063510 con 0x7ff0dc0c6f50 2024-09-17T05:27:49.504 INFO:teuthology.orchestra.run.smithi117.stdout:64424509448 2024-09-17T05:27:49.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.503+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff98c078bc0 msgr2=0x7ff98c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.503+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff98c078bc0 0x7ff98c07b0d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009b80 tx=0x7ff99c005f90 comp rx=0 tx=0).stop 2024-09-17T05:27:49.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.503+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9a815bb90 msgr2=0x7ff9a8151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.503+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9a815bb90 0x7ff9a8151cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff994031000 tx=0x7ff994003820 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.501+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8bac078b90 msgr2=0x7f8bac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.501+0000 7f8bccb84640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8bac078b90 0x7f8bac07b0a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4033000 tx=0x7f8bc0045b00 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.501+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bb8004800 msgr2=0x7f8bc016aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.501+0000 7f8bccb84640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bb8004800 0x7f8bc016aa20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc004c50 tx=0x7f8bbc001800 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 shutdown_connections 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8bac078b90 0x7f8bac07b0a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8bc00d7c80 0x7f8bc00d8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc0175a30 0x7f8bc0049840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 --2- 172.21.15.117:0/946650301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bb8004800 0x7f8bc016aa20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 >> 172.21.15.117:0/946650301 conn(0x7f8bc004fe30 msgr2=0x7f8bc0035110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 shutdown_connections 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.502+0000 7f8bccb84640 1 -- 172.21.15.117:0/946650301 wait complete. 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff0c8078bc0 msgr2=0x7ff0c807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff0c8078bc0 0x7ff0c807b0d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc000c00 tx=0x7ff0cc004730 comp rx=0 tx=0).stop 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 msgr2=0x7ff0dc15b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 0x7ff0dc15b4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009650 tx=0x7ff0d0009b10 comp rx=0 tx=0).stop 2024-09-17T05:27:49.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 shutdown_connections 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff98c078bc0 0x7ff98c07b0d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9a815bb90 0x7ff9a8151cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff9a80cc740 0x7ff9a8071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 --2- 172.21.15.117:0/1616323137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9a80cb280 0x7ff9a8070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 >> 172.21.15.117:0/1616323137 conn(0x7ff9a80744d0 msgr2=0x7ff9a80b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 shutdown_connections 2024-09-17T05:27:49.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff9ae7e6640 1 -- 172.21.15.117:0/1616323137 wait complete. 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 shutdown_connections 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff0c8078bc0 0x7ff0c807b0d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0dc0c85d0 0x7ff0dc1627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff0dc0c6f50 0x7ff0dc15b4c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 --2- 172.21.15.117:0/1367031006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0dc072e60 0x7ff0dc15cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 >> 172.21.15.117:0/1367031006 conn(0x7ff0dc0ccbe0 msgr2=0x7ff0dc0cb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.504+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 shutdown_connections 2024-09-17T05:27:49.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:49.505+0000 7ff0e149b640 1 -- 172.21.15.117:0/1367031006 wait complete. 2024-09-17T05:27:49.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.6 2024-09-17T05:27:49.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.0 2024-09-17T05:27:49.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.5 2024-09-17T05:27:49.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.3 2024-09-17T05:27:49.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.4 2024-09-17T05:27:49.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.2 2024-09-17T05:27:49.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.1 2024-09-17T05:27:49.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.7 2024-09-17T05:27:50.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T05:27:50.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T05:27:50.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T05:27:50.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T05:27:50.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T05:27:50.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T05:27:50.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T05:27:50.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T05:27:50.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.690+0000 7f994fbe6640 1 -- 172.21.15.117:0/2346806077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99340053a0 msgr2=0x7f993400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.690+0000 7f994c980640 1 -- 172.21.15.117:0/2346806077 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938037070 con 0x7f99340053a0 2024-09-17T05:27:50.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.690+0000 7f994fbe6640 1 --2- 172.21.15.117:0/2346806077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99340053a0 0x7f993400ba20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9948070010 tx=0x7f9934005830 comp rx=0 tx=0).stop 2024-09-17T05:27:50.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.690+0000 7f994e183640 1 -- 172.21.15.117:0/2346806077 reap_dead start 2024-09-17T05:27:50.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.691+0000 7f994fbe6640 1 -- 172.21.15.117:0/2346806077 shutdown_connections 2024-09-17T05:27:50.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.691+0000 7f994fbe6640 1 -- 172.21.15.117:0/2346806077 >> 172.21.15.117:0/2346806077 conn(0x7f99480ccbe0 msgr2=0x7f99480c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.692+0000 7f994fbe6640 1 -- 172.21.15.117:0/2346806077 shutdown_connections 2024-09-17T05:27:50.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.692+0000 7f994fbe6640 1 -- 172.21.15.117:0/2346806077 wait complete. 2024-09-17T05:27:50.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.695+0000 7f994fbe6640 1 Processor -- start 2024-09-17T05:27:50.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.695+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/1942067285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c0078d0 msgr2=0x7f1f2c0df6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.695+0000 7f1f3b1c3640 1 --2- 172.21.15.117:0/1942067285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c0078d0 0x7f1f2c0df6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00a600 tx=0x7f1f3002a6a0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/1942067285 shutdown_connections 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f1f3b1c3640 1 --2- 172.21.15.117:0/1942067285 >> v2:172.21.15.117:3300/0 conn(0x7f1f2c008350 0x7f1f2c04a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f1f3b1c3640 1 --2- 172.21.15.117:0/1942067285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c0078d0 0x7f1f2c0df6f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/1942067285 >> 172.21.15.117:0/1942067285 conn(0x7f1f2c050ff0 msgr2=0x7f1f2c051400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/1942067285 shutdown_connections 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/1942067285 wait complete. 2024-09-17T05:27:50.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.696+0000 7f994fbe6640 1 -- start start 2024-09-17T05:27:50.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.698+0000 7f994fbe6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99340053a0 0x7f99481e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f994fbe6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 0x7f99481e6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f994d181640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 0x7f99481e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f1f3b1c3640 1 Processor -- start 2024-09-17T05:27:50.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f994d181640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 0x7f99481e6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40288/0 (socket says 172.21.15.117:40288) 2024-09-17T05:27:50.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f1f3b1c3640 1 -- start start 2024-09-17T05:27:50.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f994d982640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99340053a0 0x7f99481e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.699+0000 7f994d982640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99340053a0 0x7f99481e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48468/0 (socket says 172.21.15.117:48468) 2024-09-17T05:27:50.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.700+0000 7f994d181640 1 -- 172.21.15.117:0/874487609 learned_addr learned my addr 172.21.15.117:0/874487609 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.700+0000 7f994fbe6640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f994815e6a0 0x7f994815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.700+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f99480733b0 con 0x7f9948152c10 2024-09-17T05:27:50.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.700+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f99480744f0 con 0x7f994815e6a0 2024-09-17T05:27:50.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.700+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f994806d4b0 con 0x7f99340053a0 2024-09-17T05:27:50.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.700+0000 7f994e183640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f994815e6a0 0x7f994815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.701+0000 7f994d181640 1 -- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99340053a0 msgr2=0x7f99481e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.701+0000 7f994d181640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99340053a0 0x7f99481e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.701+0000 7f994d181640 1 -- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f994815e6a0 msgr2=0x7f994815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.701+0000 7f994d181640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f994815e6a0 0x7f994815eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.701+0000 7f994d181640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99481e5040 con 0x7f9948152c10 2024-09-17T05:27:50.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.702+0000 7f1f3b1c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 0x7f1f2c0e24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.702+0000 7f1f3b1c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f2c007f40 0x7f1f2c179440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.702+0000 7f1f3a1c1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 0x7f1f2c0e24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.702+0000 7f1f3a1c1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 0x7f1f2c0e24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48484/0 (socket says 172.21.15.117:48484) 2024-09-17T05:27:50.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.702+0000 7f1f3a1c1640 1 -- 172.21.15.117:0/109103337 learned_addr learned my addr 172.21.15.117:0/109103337 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.702+0000 7f994d181640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 0x7f99481e6880 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99400077c0 tx=0x7f9940010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9940016070 con 0x7f9948152c10 2024-09-17T05:27:50.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f1f399c0640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f2c007f40 0x7f1f2c179440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f1f3b1c3640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c008350 0x7f1f2c0d1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99481e6dc0 con 0x7f9948152c10 2024-09-17T05:27:50.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9948153ee0 con 0x7f9948152c10 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2c048a50 con 0x7f1f2c007f40 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2c048770 con 0x7f1f2c008350 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.703+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f1f2c048d30 con 0x7f1f2c0078d0 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.704+0000 7f1f3a1c1640 1 -- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c008350 msgr2=0x7f1f2c0d1580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.704+0000 7f1f3a1c1640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c008350 0x7f1f2c0d1580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.704+0000 7f1f3a1c1640 1 -- 172.21.15.117:0/109103337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f2c007f40 msgr2=0x7f1f2c179440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.704+0000 7f1f3a1c1640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f2c007f40 0x7f1f2c179440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.704+0000 7f1f3a1c1640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2c0d7ea0 con 0x7f1f2c0078d0 2024-09-17T05:27:50.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.705+0000 7f1f399c0640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f2c007f40 0x7f1f2c179440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:50.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.705+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9940007970 con 0x7f9948152c10 2024-09-17T05:27:50.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.706+0000 7f1f3a1c1640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 0x7f1f2c0e24a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009850 tx=0x7f1f30004f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.706+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9940012430 con 0x7f9948152c10 2024-09-17T05:27:50.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.706+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9940012650 con 0x7f9948152c10 2024-09-17T05:27:50.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.707+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9930005290 con 0x7f9948152c10 2024-09-17T05:27:50.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.707+0000 7f993e7fc640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f992c078c90 0x7f992c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.708+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f994009c110 con 0x7f9948152c10 2024-09-17T05:27:50.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.708+0000 7f994d982640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f992c078c90 0x7f992c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.709+0000 7f994d982640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f992c078c90 0x7f992c07b1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9944006fd0 tx=0x7f9944008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.708+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30035030 con 0x7f1f2c0078d0 2024-09-17T05:27:50.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.708+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f30042070 con 0x7f1f2c0078d0 2024-09-17T05:27:50.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.708+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30003d10 con 0x7f1f2c0078d0 2024-09-17T05:27:50.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.709+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f2c04e420 con 0x7f1f2c0078d0 2024-09-17T05:27:50.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.709+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f2c0d7be0 con 0x7f1f2c0078d0 2024-09-17T05:27:50.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.711+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f2c0e2c80 con 0x7f1f2c0078d0 2024-09-17T05:27:50.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.713+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1f30047460 con 0x7f1f2c0078d0 2024-09-17T05:27:50.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.714+0000 7f1f2affd640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1f18078c90 0x7f1f1807b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.714+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1f300bc150 con 0x7f1f2c0078d0 2024-09-17T05:27:50.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.715+0000 7f1f399c0640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1f18078c90 0x7f1f1807b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.715+0000 7f1f399c0640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1f18078c90 0x7f1f1807b1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000f60 tx=0x7f1f24009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.716+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f99400643c0 con 0x7f9948152c10 2024-09-17T05:27:50.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.722+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1f30084230 con 0x7f1f2c0078d0 2024-09-17T05:27:50.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 -- 172.21.15.117:0/82153294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2880004b40 msgr2=0x7f28800dac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 --2- 172.21.15.117:0/82153294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2880004b40 0x7f28800dac50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f287c005fd0 tx=0x7f287c032ce0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 -- 172.21.15.117:0/82153294 shutdown_connections 2024-09-17T05:27:50.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 --2- 172.21.15.117:0/82153294 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f288004c5b0 0x7f28800469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 --2- 172.21.15.117:0/82153294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2880004b40 0x7f28800dac50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 -- 172.21.15.117:0/82153294 >> 172.21.15.117:0/82153294 conn(0x7f2880050280 msgr2=0x7f2880050690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 -- 172.21.15.117:0/82153294 shutdown_connections 2024-09-17T05:27:50.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.753+0000 7f288759e640 1 -- 172.21.15.117:0/82153294 wait complete. 2024-09-17T05:27:50.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.756+0000 7f288759e640 1 Processor -- start 2024-09-17T05:27:50.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.757+0000 7f288759e640 1 -- start start 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.759+0000 7f288759e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2880004b40 0x7f28800d5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.759+0000 7f288759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 0x7f28800d7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.759+0000 7f288759e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28800d5070 0x7f2880165e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.759+0000 7f288759e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f28800461f0 con 0x7f288004c5b0 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.759+0000 7f288759e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f28800dbb80 con 0x7f28800d5070 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.759+0000 7f288759e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2880006050 con 0x7f2880004b40 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f288659c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2880004b40 0x7f28800d5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 0x7f28800d7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 0x7f28800d7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40358/0 (socket says 172.21.15.117:40358) 2024-09-17T05:27:50.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 -- 172.21.15.117:0/2774734242 learned_addr learned my addr 172.21.15.117:0/2774734242 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2886d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28800d5070 0x7f2880165e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 -- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2880004b40 msgr2=0x7f28800d5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2880004b40 0x7f28800d5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 -- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28800d5070 msgr2=0x7f2880165e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28800d5070 0x7f2880165e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 -- 172.21.15.117:0/2774734242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2880164640 con 0x7f288004c5b0 2024-09-17T05:27:50.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.760+0000 7f2885d9b640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 0x7f28800d7900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f287800bc70 tx=0x7f28780095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.762+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287801a020 con 0x7f288004c5b0 2024-09-17T05:27:50.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28801862a0 con 0x7f288004c5b0 2024-09-17T05:27:50.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f0ace606640 1 -- 172.21.15.117:0/1778421076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac815b260 msgr2=0x7f0ac80d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f0ace606640 1 --2- 172.21.15.117:0/1778421076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac815b260 0x7f0ac80d0d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80cc8d0 tx=0x7f0ac002a8c0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2880164360 con 0x7f288004c5b0 2024-09-17T05:27:50.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f287800be80 con 0x7f288004c5b0 2024-09-17T05:27:50.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2878005020 con 0x7f288004c5b0 2024-09-17T05:27:50.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.764+0000 7feeb3990640 1 -- 172.21.15.117:0/976988601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac15ef30 msgr2=0x7feeac0c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.764+0000 7feeb3990640 1 --2- 172.21.15.117:0/976988601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac15ef30 0x7feeac0c8420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feea8005fd0 tx=0x7feea802a6a0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.764+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c005290 con 0x7f288004c5b0 2024-09-17T05:27:50.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.764+0000 7fc9d8189640 1 -- 172.21.15.117:0/3660906226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9bc005300 msgr2=0x7fc9bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.764+0000 7fc9d8189640 1 --2- 172.21.15.117:0/3660906226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9bc005300 0x7fc9bc00b9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800dcb0 tx=0x7fc9bc005830 comp rx=0 tx=0).stop 2024-09-17T05:27:50.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7feeb3990640 1 -- 172.21.15.117:0/976988601 shutdown_connections 2024-09-17T05:27:50.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7feeb3990640 1 --2- 172.21.15.117:0/976988601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac15ef30 0x7feeac0c8420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7feeb3990640 1 --2- 172.21.15.117:0/976988601 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feeac073930 0x7feeac15e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7feeb3990640 1 -- 172.21.15.117:0/976988601 >> 172.21.15.117:0/976988601 conn(0x7feeac0c9a20 msgr2=0x7feeac0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7fc9d6726640 1 -- 172.21.15.117:0/3660906226 reap_dead start 2024-09-17T05:27:50.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7f56c0664640 1 -- 172.21.15.117:0/3772100855 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 msgr2=0x7f56b8152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7fc9d8189640 1 -- 172.21.15.117:0/3660906226 shutdown_connections 2024-09-17T05:27:50.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7fc9d8189640 1 -- 172.21.15.117:0/3660906226 >> 172.21.15.117:0/3660906226 conn(0x7fc9d00ca0e0 msgr2=0x7fc9d00710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7feeb3990640 1 -- 172.21.15.117:0/976988601 shutdown_connections 2024-09-17T05:27:50.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7feeb3990640 1 -- 172.21.15.117:0/976988601 wait complete. 2024-09-17T05:27:50.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7f56c0664640 1 --2- 172.21.15.117:0/3772100855 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 0x7f56b8152210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009860 tx=0x7f56a802a460 comp rx=0 tx=0).stop 2024-09-17T05:27:50.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f0ace606640 1 -- 172.21.15.117:0/1778421076 shutdown_connections 2024-09-17T05:27:50.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f0ace606640 1 --2- 172.21.15.117:0/1778421076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac815b260 0x7f0ac80d0d60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f0ace606640 1 --2- 172.21.15.117:0/1778421076 >> v2:172.21.15.117:3300/0 conn(0x7f0ac80ca840 0x7f0ac8155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.763+0000 7f0ace606640 1 -- 172.21.15.117:0/1778421076 >> 172.21.15.117:0/1778421076 conn(0x7f0ac8075e00 msgr2=0x7f0ac8076210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7f0ace606640 1 -- 172.21.15.117:0/1778421076 shutdown_connections 2024-09-17T05:27:50.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7fc9d8189640 1 -- 172.21.15.117:0/3660906226 shutdown_connections 2024-09-17T05:27:50.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.765+0000 7fc9d8189640 1 -- 172.21.15.117:0/3660906226 wait complete. 2024-09-17T05:27:50.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.766+0000 7f0ace606640 1 -- 172.21.15.117:0/1778421076 wait complete. 2024-09-17T05:27:50.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.767+0000 7f56c0664640 1 -- 172.21.15.117:0/3772100855 shutdown_connections 2024-09-17T05:27:50.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.767+0000 7f56c0664640 1 --2- 172.21.15.117:0/3772100855 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 0x7f56b8152210 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.767+0000 7f56c0664640 1 -- 172.21.15.117:0/3772100855 >> 172.21.15.117:0/3772100855 conn(0x7f56b806e590 msgr2=0x7f56b80cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.767+0000 7f56c0664640 1 -- 172.21.15.117:0/3772100855 shutdown_connections 2024-09-17T05:27:50.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7fc9d8189640 1 Processor -- start 2024-09-17T05:27:50.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7feeb3990640 1 Processor -- start 2024-09-17T05:27:50.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7fc9d8189640 1 -- start start 2024-09-17T05:27:50.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7feeb3990640 1 -- start start 2024-09-17T05:27:50.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f28780051c0 con 0x7f288004c5b0 2024-09-17T05:27:50.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.769+0000 7f0ace606640 1 Processor -- start 2024-09-17T05:27:50.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7f2876ffd640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2868078bc0 0x7f286807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.768+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f287809bc10 con 0x7f288004c5b0 2024-09-17T05:27:50.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.767+0000 7f56c0664640 1 -- 172.21.15.117:0/3772100855 wait complete. 2024-09-17T05:27:50.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.769+0000 7f0ace606640 1 -- start start 2024-09-17T05:27:50.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7f288659c640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2868078bc0 0x7f286807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d8189640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 0x7fc9d01ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d8189640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d0155b90 0x7fc9d006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d8189640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d0155470 0x7fc9d006d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7f56c0664640 1 Processor -- start 2024-09-17T05:27:50.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d8189640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d00c9730 con 0x7fc9d0155470 2024-09-17T05:27:50.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d8189640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d00c3910 con 0x7fc9bc005300 2024-09-17T05:27:50.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7f56c0664640 1 -- start start 2024-09-17T05:27:50.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d8189640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d0071ed0 con 0x7fc9d0155b90 2024-09-17T05:27:50.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d5f25640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 0x7fc9d01ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d6726640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d0155470 0x7fc9d006d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d5f25640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 0x7fc9d01ee1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44164/0 (socket says 172.21.15.117:44164) 2024-09-17T05:27:50.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.770+0000 7fc9d5f25640 1 -- 172.21.15.117:0/2862269231 learned_addr learned my addr 172.21.15.117:0/2862269231 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7feeb3990640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feeac073930 0x7feeac152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d5724640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d0155b90 0x7fc9d006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7feeb3990640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 0x7feeac1562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d6726640 1 -- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d0155b90 msgr2=0x7fc9d006cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d6726640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d0155b90 0x7fc9d006cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d6726640 1 -- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 msgr2=0x7fc9d01ee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d6726640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 0x7fc9d01ee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d6726640 1 -- 172.21.15.117:0/2862269231 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d01ee720 con 0x7fc9d0155470 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d5f25640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 0x7fc9d01ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:50.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d5724640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d0155b90 0x7fc9d006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:50.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7f0ace606640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac80b4c40 0x7f0ac8157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7f0ace606640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 0x7f0ac8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7feeb0f2b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 0x7feeac1562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7feeb0f2b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 0x7feeac1562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40392/0 (socket says 172.21.15.117:40392) 2024-09-17T05:27:50.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7fc9d6726640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d0155470 0x7fc9d006d2e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00c230 tx=0x7fc9c8036850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.771+0000 7feeb0f2b640 1 -- 172.21.15.117:0/2278491437 learned_addr learned my addr 172.21.15.117:0/2278491437 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeac15ef30 0x7feeac15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 0x7f0ac8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 0x7f0ac8156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40398/0 (socket says 172.21.15.117:40398) 2024-09-17T05:27:50.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 -- 172.21.15.117:0/3330625087 learned_addr learned my addr 172.21.15.117:0/3330625087 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f288659c640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2868078bc0 0x7f286807b0d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f287c03c000 tx=0x7f287c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ac815b260 0x7f0ac8150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac7fff640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac80b4c40 0x7f0ac8157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac80cc100 con 0x7f0ac80ca840 2024-09-17T05:27:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac80cbe20 con 0x7f0ac80b4c40 2024-09-17T05:27:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac80cc3e0 con 0x7f0ac815b260 2024-09-17T05:27:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7feeb172c640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feeac073930 0x7feeac152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feeac0c8d70 con 0x7feeac0c8960 2024-09-17T05:27:50.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0accba3640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ac815b260 0x7f0ac8150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7feeb1f2d640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeac15ef30 0x7feeac15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feeac074930 con 0x7feeac15ef30 2024-09-17T05:27:50.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7feeac15f6a0 con 0x7feeac073930 2024-09-17T05:27:50.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 -- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ac815b260 msgr2=0x7f0ac8150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ac815b260 0x7f0ac8150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 -- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac80b4c40 msgr2=0x7f0ac8157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac80b4c40 0x7f0ac8157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.772+0000 7f0ac77fe640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac81583e0 con 0x7f0ac80ca840 2024-09-17T05:27:50.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56c0664640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 0x7f56b81ed560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f0accba3640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ac815b260 0x7f0ac8150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:50.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f0ac77fe640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 0x7f0ac8156b50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00b410 tx=0x7f0abc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56c0664640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56b80c5660 0x7f56b81e7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 0x7f56b81ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 0x7f56b81ed560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40414/0 (socket says 172.21.15.117:40414) 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 -- 172.21.15.117:0/1928422911 learned_addr learned my addr 172.21.15.117:0/1928422911 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7feeb0f2b640 1 -- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feeac073930 msgr2=0x7feeac152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7feeb0f2b640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feeac073930 0x7feeac152370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7feeb0f2b640 1 -- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeac15ef30 msgr2=0x7feeac15af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7feeb0f2b640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeac15ef30 0x7feeac15af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7feeb0f2b640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac1f9bd0 con 0x7feeac0c8960 2024-09-17T05:27:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 0x7f56b814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f56b80c7ff0 con 0x7f56b8071850 2024-09-17T05:27:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f56b80ce0d0 con 0x7f56b80c5660 2024-09-17T05:27:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f56b80c83a0 con 0x7f56b80c67a0 2024-09-17T05:27:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56bdbff640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56b80c5660 0x7f56b81e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7feeb0f2b640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 0x7feeac1562e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feea8034040 tx=0x7feea8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 -- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 msgr2=0x7f56b814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 0x7f56b814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 -- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56b80c5660 msgr2=0x7f56b81e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.773+0000 7f56be400640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56b80c5660 0x7f56b81e7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f56be400640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b81f1a90 con 0x7f56b8071850 2024-09-17T05:27:50.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc015020 con 0x7f0ac80ca840 2024-09-17T05:27:50.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac8157090 con 0x7f0ac80ca840 2024-09-17T05:27:50.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f56be400640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 0x7f56b81ed560 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f56a8006010 tx=0x7f56a8033f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2878024020 con 0x7f288004c5b0 2024-09-17T05:27:50.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac80cf820 con 0x7f0ac80ca840 2024-09-17T05:27:50.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea803e030 con 0x7feeac0c8960 2024-09-17T05:27:50.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feea80047a0 con 0x7feeac0c8960 2024-09-17T05:27:50.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c800d8b0 con 0x7fc9d0155470 2024-09-17T05:27:50.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d00cca70 con 0x7fc9d0155470 2024-09-17T05:27:50.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac1fa150 con 0x7feeac0c8960 2024-09-17T05:27:50.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d015a710 con 0x7fc9d0155470 2024-09-17T05:27:50.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9c8004b20 con 0x7fc9d0155470 2024-09-17T05:27:50.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8036b50 con 0x7fc9d0155470 2024-09-17T05:27:50.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0abc011830 con 0x7f0ac80ca840 2024-09-17T05:27:50.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.774+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc011c20 con 0x7f0ac80ca840 2024-09-17T05:27:50.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.776+0000 7f4f61332640 1 -- 172.21.15.117:0/2401908390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c073f60 msgr2=0x7f4f5c0c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.776+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa8005290 con 0x7f0ac80ca840 2024-09-17T05:27:50.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.776+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8031e20 con 0x7f56b8071850 2024-09-17T05:27:50.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.776+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56a80331c0 con 0x7f56b8071850 2024-09-17T05:27:50.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.777+0000 7f4f59ffb640 1 -- 172.21.15.117:0/2401908390 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500047a0 con 0x7f4f5c073f60 2024-09-17T05:27:50.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.777+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a80335f0 con 0x7f56b8071850 2024-09-17T05:27:50.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.777+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b8153d80 con 0x7f56b8071850 2024-09-17T05:27:50.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.777+0000 7f4f61332640 1 --2- 172.21.15.117:0/2401908390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c073f60 0x7f4f5c0c8390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4f50005fd0 tx=0x7f4f5002a540 comp rx=0 tx=0).stop 2024-09-17T05:27:50.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.777+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b8154020 con 0x7f56b8071850 2024-09-17T05:27:50.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.778+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0abc018020 con 0x7f0ac80ca840 2024-09-17T05:27:50.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f4f61332640 1 -- 172.21.15.117:0/2401908390 shutdown_connections 2024-09-17T05:27:50.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f4f61332640 1 --2- 172.21.15.117:0/2401908390 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c0b2740 0x7f4f5c0b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f4f61332640 1 --2- 172.21.15.117:0/2401908390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c073f60 0x7f4f5c0c8390 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f4f61332640 1 -- 172.21.15.117:0/2401908390 >> 172.21.15.117:0/2401908390 conn(0x7f4f5c0c9a20 msgr2=0x7f4f5c0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:50.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.778+0000 7f0ac4ff9640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0aac078b90 0x7f0aac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0abc09b4a0 con 0x7f0ac80ca840 2024-09-17T05:27:50.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9c8036070 con 0x7fc9d0155470 2024-09-17T05:27:50.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8035040 con 0x7feeac0c8960 2024-09-17T05:27:50.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f0ac7fff640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0aac078b90 0x7f0aac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d006df80 con 0x7fc9d0155470 2024-09-17T05:27:50.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7fc9c67fc640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b4078bc0 0x7fc9b407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc9c80bf4e0 con 0x7fc9d0155470 2024-09-17T05:27:50.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f4f61332640 1 -- 172.21.15.117:0/2401908390 shutdown_connections 2024-09-17T05:27:50.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.779+0000 7f4f61332640 1 -- 172.21.15.117:0/2401908390 wait complete. 2024-09-17T05:27:50.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7f0ac7fff640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0aac078b90 0x7f0aac07b0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4007fd0 tx=0x7f0ab4001370 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.781+0000 7fc9d5f25640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b4078bc0 0x7fc9b407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f56a8003670 con 0x7f56b8071850 2024-09-17T05:27:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b8072510 con 0x7f56b8071850 2024-09-17T05:27:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.781+0000 7f56aeffd640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f569c078c60 0x7f569c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.781+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56a80bc580 con 0x7f56b8071850 2024-09-17T05:27:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac1518d0 con 0x7feeac0c8960 2024-09-17T05:27:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.780+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feea8041030 con 0x7feeac0c8960 2024-09-17T05:27:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.782+0000 7feea1ffb640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fee90078c10 0x7fee9007b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.782+0000 7fc9d5f25640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b4078bc0 0x7fc9b407b0d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc000fb0 tx=0x7fc9cc006040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.782+0000 7f56bdbff640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f569c078c60 0x7f569c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.782+0000 7feeb172c640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fee90078c10 0x7fee9007b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.783+0000 7f4f61332640 1 Processor -- start 2024-09-17T05:27:50.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.783+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee8c005290 con 0x7feeac0c8960 2024-09-17T05:27:50.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.783+0000 7f4f61332640 1 -- start start 2024-09-17T05:27:50.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.783+0000 7feeb172c640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fee90078c10 0x7fee9007b120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006fd0 tx=0x7fee9c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.784+0000 7f56bdbff640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f569c078c60 0x7f569c07b170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f56b814e600 tx=0x7f56b4007e90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.785+0000 7f4f61332640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 0x7f4f5c1525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.785+0000 7f4f61332640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 0x7f4f5c148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.785+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0abc01f020 con 0x7f0ac80ca840 2024-09-17T05:27:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5affd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 0x7f4f5c1525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5affd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 0x7f4f5c1525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48592/0 (socket says 172.21.15.117:48592) 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5a7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 0x7f4f5c148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5a7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 0x7f4f5c148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40424/0 (socket says 172.21.15.117:40424) 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5affd640 1 -- 172.21.15.117:0/3526356194 learned_addr learned my addr 172.21.15.117:0/3526356194 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f5c0c6f50 0x7f4f5c154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5c157af0 con 0x7f4f5c0b2740 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5c157810 con 0x7f4f5c0c6f50 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5c157dd0 con 0x7f4f5c073f60 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5b7fe640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f5c0c6f50 0x7f4f5c154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5affd640 1 -- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f5c0c6f50 msgr2=0x7f4f5c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5affd640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f5c0c6f50 0x7f4f5c154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.786+0000 7f4f5affd640 1 -- 172.21.15.117:0/3526356194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 msgr2=0x7f4f5c148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f5affd640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 0x7f4f5c148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f5affd640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5c152af0 con 0x7f4f5c073f60 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f5b7fe640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f5c0c6f50 0x7f4f5c154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f5a7fc640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 0x7f4f5c148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f5affd640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 0x7f4f5c1525b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009620 tx=0x7f4f4c009ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00e8f0 con 0x7f4f5c073f60 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f4c00ea90 con 0x7f4f5c073f60 2024-09-17T05:27:50.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.787+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c017400 con 0x7f4f5c073f60 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.788+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f5c14ea90 con 0x7f4f5c073f60 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.788+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f5c14f050 con 0x7f4f5c073f60 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.789+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f5c14f290 con 0x7f4f5c073f60 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.790+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4f4c00f400 con 0x7f4f5c073f60 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.791+0000 7f4f437fe640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4f48078b90 0x7f4f4807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.791+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4f4c09b040 con 0x7f4f5c073f60 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.792+0000 7f4f5a7fc640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4f48078b90 0x7f4f4807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:50.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.793+0000 7f4f5a7fc640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4f48078b90 0x7f4f4807b0a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f50033000 tx=0x7f4f50004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:50.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.795+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9d006df80 con 0x7fc9d0155470 2024-09-17T05:27:50.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.795+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7feea807f020 con 0x7feeac0c8960 2024-09-17T05:27:50.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.797+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4f5c14f290 con 0x7f4f5c073f60 2024-09-17T05:27:50.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.805+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f56b8072510 con 0x7f56b8071850 2024-09-17T05:27:50.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:50.815+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feeac1f9bd0 con 0x7feeac0c8960 2024-09-17T05:27:51.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.119+0000 7f994fbe6640 1 -- 172.21.15.117:0/874487609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f9930005580 con 0x7f9948152c10 2024-09-17T05:27:51.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.119+0000 7f1f3b1c3640 1 -- 172.21.15.117:0/109103337 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1f2c0492a0 con 0x7f1f2c0078d0 2024-09-17T05:27:51.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.120+0000 7f1f2affd640 1 -- 172.21.15.117:0/109103337 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1f30083b30 con 0x7f1f2c0078d0 2024-09-17T05:27:51.121 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:51.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.120+0000 7f993e7fc640 1 -- 172.21.15.117:0/874487609 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9940063cc0 con 0x7f9948152c10 2024-09-17T05:27:51.122 INFO:teuthology.orchestra.run.smithi117.stdout:60129542154 2024-09-17T05:27:51.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.123+0000 7f1f28ff9640 1 -- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1f18078c90 msgr2=0x7f1f1807b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.123+0000 7f1f28ff9640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1f18078c90 0x7f1f1807b1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000f60 tx=0x7f1f24009040 comp rx=0 tx=0).stop 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 -- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 msgr2=0x7f1f2c0e24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 0x7f1f2c0e24a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009850 tx=0x7f1f30004f60 comp rx=0 tx=0).stop 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 -- 172.21.15.117:0/109103337 shutdown_connections 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1f18078c90 0x7f1f1807b1a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2c008350 0x7f1f2c0d1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f2c007f40 0x7f1f2c179440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 --2- 172.21.15.117:0/109103337 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1f2c0078d0 0x7f1f2c0e24a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 -- 172.21.15.117:0/109103337 >> 172.21.15.117:0/109103337 conn(0x7f1f2c050ff0 msgr2=0x7f1f2c0dd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 -- 172.21.15.117:0/109103337 shutdown_connections 2024-09-17T05:27:51.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f1f28ff9640 1 -- 172.21.15.117:0/109103337 wait complete. 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f992bfff640 1 -- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f992c078c90 msgr2=0x7f992c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.124+0000 7f992bfff640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f992c078c90 0x7f992c07b1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9944006fd0 tx=0x7f9944008040 comp rx=0 tx=0).stop 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 -- 172.21.15.117:0/874487609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 msgr2=0x7f99481e6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 0x7f99481e6880 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99400077c0 tx=0x7f9940010720 comp rx=0 tx=0).stop 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 -- 172.21.15.117:0/874487609 shutdown_connections 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f992c078c90 0x7f992c07b1a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f994815e6a0 0x7f994815eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948152c10 0x7f99481e6880 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 --2- 172.21.15.117:0/874487609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99340053a0 0x7f99481e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.125+0000 7f992bfff640 1 -- 172.21.15.117:0/874487609 >> 172.21.15.117:0/874487609 conn(0x7f99480ccbe0 msgr2=0x7f9948068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.126+0000 7f992bfff640 1 -- 172.21.15.117:0/874487609 shutdown_connections 2024-09-17T05:27:51.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.126+0000 7f992bfff640 1 -- 172.21.15.117:0/874487609 wait complete. 2024-09-17T05:27:51.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.6 2024-09-17T05:27:51.145 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.0 2024-09-17T05:27:51.145 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.168+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f286c005850 con 0x7f288004c5b0 2024-09-17T05:27:51.170 INFO:teuthology.orchestra.run.smithi117.stdout:60129542154 2024-09-17T05:27:51.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.169+0000 7f2876ffd640 1 -- 172.21.15.117:0/2774734242 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2878063e10 con 0x7f288004c5b0 2024-09-17T05:27:51.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.170+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc9d00cc800 con 0x7fc9d0155470 2024-09-17T05:27:51.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.170+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f4f5c0c94d0 con 0x7f4f5c073f60 2024-09-17T05:27:51.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.170+0000 7fc9c67fc640 1 -- 172.21.15.117:0/2862269231 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc9c804a060 con 0x7fc9d0155470 2024-09-17T05:27:51.172 INFO:teuthology.orchestra.run.smithi117.stdout:64424509450 2024-09-17T05:27:51.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.170+0000 7f4f437fe640 1 -- 172.21.15.117:0/3526356194 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4f4c013030 con 0x7f4f5c073f60 2024-09-17T05:27:51.172 INFO:teuthology.orchestra.run.smithi117.stdout:60129542154 2024-09-17T05:27:51.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2868078bc0 msgr2=0x7f286807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2868078bc0 0x7f286807b0d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f287c03c000 tx=0x7f287c004730 comp rx=0 tx=0).stop 2024-09-17T05:27:51.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 msgr2=0x7f28800d7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 0x7f28800d7900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f287800bc70 tx=0x7f28780095b0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 shutdown_connections 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2868078bc0 0x7f286807b0d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28800d5070 0x7f2880165e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f288004c5b0 0x7f28800d7900 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.172+0000 7f288759e640 1 --2- 172.21.15.117:0/2774734242 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2880004b40 0x7f28800d5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b4078bc0 msgr2=0x7fc9b407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b4078bc0 0x7fc9b407b0d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc000fb0 tx=0x7fc9cc006040 comp rx=0 tx=0).stop 2024-09-17T05:27:51.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 >> 172.21.15.117:0/2774734242 conn(0x7f2880050280 msgr2=0x7f28800d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d0155470 msgr2=0x7fc9d006d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d0155470 0x7fc9d006d2e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00c230 tx=0x7fc9c8036850 comp rx=0 tx=0).stop 2024-09-17T05:27:51.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 shutdown_connections 2024-09-17T05:27:51.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f288759e640 1 -- 172.21.15.117:0/2774734242 wait complete. 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 shutdown_connections 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b4078bc0 0x7fc9b407b0d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d0155470 0x7fc9d006d2e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d0155b90 0x7fc9d006cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 --2- 172.21.15.117:0/2862269231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9bc005300 0x7fc9d01ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 >> 172.21.15.117:0/2862269231 conn(0x7fc9d00ca0e0 msgr2=0x7fc9d014e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 shutdown_connections 2024-09-17T05:27:51.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7fc9d8189640 1 -- 172.21.15.117:0/2862269231 wait complete. 2024-09-17T05:27:51.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4f48078b90 msgr2=0x7f4f4807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4f48078b90 0x7f4f4807b0a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4f50033000 tx=0x7f4f50004730 comp rx=0 tx=0).stop 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 msgr2=0x7f4f5c1525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 0x7f4f5c1525b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009620 tx=0x7f4f4c009ae0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 shutdown_connections 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4f48078b90 0x7f4f4807b0a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f5c0c6f50 0x7f4f5c154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f5c0b2740 0x7f4f5c148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 --2- 172.21.15.117:0/3526356194 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f5c073f60 0x7f4f5c1525b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.173+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 >> 172.21.15.117:0/3526356194 conn(0x7f4f5c0c9a20 msgr2=0x7f4f5c151270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.174+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 shutdown_connections 2024-09-17T05:27:51.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.174+0000 7f4f61332640 1 -- 172.21.15.117:0/3526356194 wait complete. 2024-09-17T05:27:51.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.4 2024-09-17T05:27:51.193 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.2 2024-09-17T05:27:51.193 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.194 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.7 2024-09-17T05:27:51.194 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.202+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f56b8157f10 con 0x7f56b8071850 2024-09-17T05:27:51.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.202+0000 7f56aeffd640 1 -- 172.21.15.117:0/1928422911 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f56a803f030 con 0x7f56b8071850 2024-09-17T05:27:51.204 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:51.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.204+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0aa8005850 con 0x7f0ac80ca840 2024-09-17T05:27:51.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.204+0000 7f0ac4ff9640 1 -- 172.21.15.117:0/3330625087 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0abc063050 con 0x7f0ac80ca840 2024-09-17T05:27:51.205 INFO:teuthology.orchestra.run.smithi117.stdout:55834574858 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f569c078c60 msgr2=0x7f569c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f569c078c60 0x7f569c07b170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f56b814e600 tx=0x7f56b4007e90 comp rx=0 tx=0).stop 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 msgr2=0x7f56b81ed560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 0x7f56b81ed560 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f56a8006010 tx=0x7f56a8033f10 comp rx=0 tx=0).stop 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 shutdown_connections 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f569c078c60 0x7f569c07b170 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f56b80c67a0 0x7f56b814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56b80c5660 0x7f56b81e7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 --2- 172.21.15.117:0/1928422911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56b8071850 0x7f56b81ed560 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 >> 172.21.15.117:0/1928422911 conn(0x7f56b806e590 msgr2=0x7f56b8158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 shutdown_connections 2024-09-17T05:27:51.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.205+0000 7f56c0664640 1 -- 172.21.15.117:0/1928422911 wait complete. 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0aac078b90 msgr2=0x7f0aac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0aac078b90 0x7f0aac07b0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4007fd0 tx=0x7f0ab4001370 comp rx=0 tx=0).stop 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 msgr2=0x7f0ac8156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 0x7f0ac8156b50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00b410 tx=0x7f0abc00b8d0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 shutdown_connections 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0aac078b90 0x7f0aac07b0a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ac815b260 0x7f0ac8150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ac80ca840 0x7f0ac8156b50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 --2- 172.21.15.117:0/3330625087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ac80b4c40 0x7f0ac8157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 >> 172.21.15.117:0/3330625087 conn(0x7f0ac8075e00 msgr2=0x7f0ac8159290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 shutdown_connections 2024-09-17T05:27:51.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.206+0000 7f0ace606640 1 -- 172.21.15.117:0/3330625087 wait complete. 2024-09-17T05:27:51.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.1 2024-09-17T05:27:51.219 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.224 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2024-09-17T05:27:51.224 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.225+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fee8c005850 con 0x7feeac0c8960 2024-09-17T05:27:51.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.226+0000 7feea1ffb640 1 -- 172.21.15.117:0/2278491437 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feea8082db0 con 0x7feeac0c8960 2024-09-17T05:27:51.228 INFO:teuthology.orchestra.run.smithi117.stdout:64424509449 2024-09-17T05:27:51.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fee90078c10 msgr2=0x7fee9007b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fee90078c10 0x7fee9007b120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006fd0 tx=0x7fee9c008040 comp rx=0 tx=0).stop 2024-09-17T05:27:51.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 msgr2=0x7feeac1562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 0x7feeac1562e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feea8034040 tx=0x7feea8003ba0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 shutdown_connections 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fee90078c10 0x7fee9007b120 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeac15ef30 0x7feeac15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeac0c8960 0x7feeac1562e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 --2- 172.21.15.117:0/2278491437 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feeac073930 0x7feeac152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 >> 172.21.15.117:0/2278491437 conn(0x7feeac0c9a20 msgr2=0x7feeac0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 shutdown_connections 2024-09-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.229+0000 7feeb3990640 1 -- 172.21.15.117:0/2278491437 wait complete. 2024-09-17T05:27:51.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.3 2024-09-17T05:27:51.241 DEBUG:teuthology.parallel:result is None 2024-09-17T05:27:51.241 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T05:27:51.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 -- 172.21.15.117:0/732810100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec1581e0 msgr2=0x7fa1ec1516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 --2- 172.21.15.117:0/732810100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec1581e0 0x7fa1ec1516f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec072fc0 tx=0x7fa1dc02b930 comp rx=0 tx=0).stop 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 -- 172.21.15.117:0/732810100 shutdown_connections 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 --2- 172.21.15.117:0/732810100 >> v2:172.21.15.119:3300/0 conn(0x7fa1ec070890 0x7fa1ec070ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 --2- 172.21.15.117:0/732810100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec1581e0 0x7fa1ec1516f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 -- 172.21.15.117:0/732810100 >> 172.21.15.117:0/732810100 conn(0x7fa1ec06d190 msgr2=0x7fa1ec06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 -- 172.21.15.117:0/732810100 shutdown_connections 2024-09-17T05:27:51.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.392+0000 7fa1f1113640 1 -- 172.21.15.117:0/732810100 wait complete. 2024-09-17T05:27:51.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.394+0000 7fa1f1113640 1 Processor -- start 2024-09-17T05:27:51.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.394+0000 7fa1f1113640 1 -- start start 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1f1113640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 0x7fa1ec1e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1f1113640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1ec0c8990 0x7fa1ec073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1f1113640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1ec1581e0 0x7fa1ec0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1f1113640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0c94c0 con 0x7fa1ec070890 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1f1113640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec0c9340 con 0x7fa1ec0c8990 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1f1113640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec0c9640 con 0x7fa1ec1581e0 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.395+0000 7fa1ead9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 0x7fa1ec1e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 0x7fa1ec1e4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40450/0 (socket says 172.21.15.117:40450) 2024-09-17T05:27:51.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 -- 172.21.15.117:0/2495851244 learned_addr learned my addr 172.21.15.117:0/2495851244 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1eb59e640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1ec1581e0 0x7fa1ec0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1e3fff640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1ec0c8990 0x7fa1ec073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 -- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1ec1581e0 msgr2=0x7fa1ec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1ec1581e0 0x7fa1ec0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 -- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1ec0c8990 msgr2=0x7fa1ec073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1ec0c8990 0x7fa1ec073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 -- 172.21.15.117:0/2495851244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec1e4670 con 0x7fa1ec070890 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1ead9d640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 0x7fa1ec1e4130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00aad0 tx=0x7fa1dc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc03e070 con 0x7fa1ec070890 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1dc004cc0 con 0x7fa1ec070890 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec15b400 con 0x7fa1ec070890 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc005d50 con 0x7fa1ec070890 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.396+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec15a5e0 con 0x7fa1ec070890 2024-09-17T05:27:51.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.397+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec1516f0 con 0x7fa1ec070890 2024-09-17T05:27:51.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.398+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa1dc038070 con 0x7fa1ec070890 2024-09-17T05:27:51.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.398+0000 7fa1e37fe640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa1cc078b90 0x7fa1cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.398+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa1dc0bbd20 con 0x7fa1ec070890 2024-09-17T05:27:51.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.399+0000 7fa1e3fff640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa1cc078b90 0x7fa1cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.399+0000 7fa1e3fff640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa1cc078b90 0x7fa1cc07b0a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4000f60 tx=0x7fa1d4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:51.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.403+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa1ec1e4670 con 0x7fa1ec070890 2024-09-17T05:27:51.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.573+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1ec0c4dd0 con 0x7fa1cc078b90 2024-09-17T05:27:51.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.579+0000 7fa1e37fe640 1 -- 172.21.15.117:0/2495851244 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332736 (secure 0 0 0) 0x7fa1ec0c4dd0 con 0x7fa1cc078b90 2024-09-17T05:27:51.584 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:51.585 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa1cc078b90 msgr2=0x7fa1cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa1cc078b90 0x7fa1cc07b0a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4000f60 tx=0x7fa1d4009040 comp rx=0 tx=0).stop 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 msgr2=0x7fa1ec1e4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 0x7fa1ec1e4130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00aad0 tx=0x7fa1dc004360 comp rx=0 tx=0).stop 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 shutdown_connections 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa1cc078b90 0x7fa1cc07b0a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1ec1581e0 0x7fa1ec0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1ec0c8990 0x7fa1ec073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 --2- 172.21.15.117:0/2495851244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1ec070890 0x7fa1ec1e4130 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 >> 172.21.15.117:0/2495851244 conn(0x7fa1ec06d190 msgr2=0x7fa1ec14c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 shutdown_connections 2024-09-17T05:27:51.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.582+0000 7fa1f1113640 1 -- 172.21.15.117:0/2495851244 wait complete. 2024-09-17T05:27:51.595 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-09-17T05:27:51.355357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595792,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":186,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218436,"kb_used_data":2232,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756284,"statfs":{"total":773094113280,"available":772870434816,"internally_reserved":0,"allocated":2285568,"data_stored":2007926,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3014,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004049"},"pg_stats":[{"pgid":"2.1f","version":"24'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688102+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:31.688102+0000","last_peered":"2024-09-17T05:27:31.688102+0000","last_clean":"2024-09-17T05:27:31.688102+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:31.688102+0000","last_undegraded":"2024-09-17T05:27:31.688102+0000","last_fullsized":"2024-09-17T05:27:31.688102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688831+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688831+0000","last_peered":"2024-09-17T05:27:31.688831+0000","last_clean":"2024-09-17T05:27:31.688831+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688831+0000","last_undegraded":"2024-09-17T05:27:31.688831+0000","last_fullsized":"2024-09-17T05:27:31.688831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490251+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:32.490251+0000","last_peered":"2024-09-17T05:27:32.490251+0000","last_clean":"2024-09-17T05:27:32.490251+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:32.490251+0000","last_undegraded":"2024-09-17T05:27:32.490251+0000","last_fullsized":"2024-09-17T05:27:32.490251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749303+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:48.749303+0000","last_peered":"2024-09-17T05:27:48.749303+0000","last_clean":"2024-09-17T05:27:48.749303+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:48.749303+0000","last_undegraded":"2024-09-17T05:27:48.749303+0000","last_fullsized":"2024-09-17T05:27:48.749303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686433+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686433+0000","last_peered":"2024-09-17T05:27:31.686433+0000","last_clean":"2024-09-17T05:27:31.686433+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686433+0000","last_undegraded":"2024-09-17T05:27:31.686433+0000","last_fullsized":"2024-09-17T05:27:31.686433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747002+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:48.747002+0000","last_peered":"2024-09-17T05:27:48.747002+0000","last_clean":"2024-09-17T05:27:48.747002+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:48.747002+0000","last_undegraded":"2024-09-17T05:27:48.747002+0000","last_fullsized":"2024-09-17T05:27:48.747002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.748010+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:48.748010+0000","last_peered":"2024-09-17T05:27:48.748010+0000","last_clean":"2024-09-17T05:27:48.748010+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:48.748010+0000","last_undegraded":"2024-09-17T05:27:48.748010+0000","last_fullsized":"2024-09-17T05:27:48.748010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750067+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:48.750067+0000","last_peered":"2024-09-17T05:27:48.750067+0000","last_clean":"2024-09-17T05:27:48.750067+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:48.750067+0000","last_undegraded":"2024-09-17T05:27:48.750067+0000","last_fullsized":"2024-09-17T05:27:48.750067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686707+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:31.686707+0000","last_peered":"2024-09-17T05:27:31.686707+0000","last_clean":"2024-09-17T05:27:31.686707+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:31.686707+0000","last_undegraded":"2024-09-17T05:27:31.686707+0000","last_fullsized":"2024-09-17T05:27:31.686707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490238+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490238+0000","last_peered":"2024-09-17T05:27:32.490238+0000","last_clean":"2024-09-17T05:27:32.490238+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490238+0000","last_undegraded":"2024-09-17T05:27:32.490238+0000","last_fullsized":"2024-09-17T05:27:32.490238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490613+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490613+0000","last_peered":"2024-09-17T05:27:32.490613+0000","last_clean":"2024-09-17T05:27:32.490613+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490613+0000","last_undegraded":"2024-09-17T05:27:32.490613+0000","last_fullsized":"2024-09-17T05:27:32.490613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"25'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488421+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:32.488421+0000","last_peered":"2024-09-17T05:27:32.488421+0000","last_clean":"2024-09-17T05:27:32.488421+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:32.488421+0000","last_undegraded":"2024-09-17T05:27:32.488421+0000","last_fullsized":"2024-09-17T05:27:32.488421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489023+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:32.489023+0000","last_peered":"2024-09-17T05:27:32.489023+0000","last_clean":"2024-09-17T05:27:32.489023+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:32.489023+0000","last_undegraded":"2024-09-17T05:27:32.489023+0000","last_fullsized":"2024-09-17T05:27:32.489023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"25'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749871+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:48.749871+0000","last_peered":"2024-09-17T05:27:48.749871+0000","last_clean":"2024-09-17T05:27:48.749871+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:48.749871+0000","last_undegraded":"2024-09-17T05:27:48.749871+0000","last_fullsized":"2024-09-17T05:27:48.749871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687718+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:31.687718+0000","last_peered":"2024-09-17T05:27:31.687718+0000","last_clean":"2024-09-17T05:27:31.687718+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:31.687718+0000","last_undegraded":"2024-09-17T05:27:31.687718+0000","last_fullsized":"2024-09-17T05:27:31.687718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489640+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489640+0000","last_peered":"2024-09-17T05:27:32.489640+0000","last_clean":"2024-09-17T05:27:32.489640+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489640+0000","last_undegraded":"2024-09-17T05:27:32.489640+0000","last_fullsized":"2024-09-17T05:27:32.489640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"25'10","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.742564+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:48.742564+0000","last_peered":"2024-09-17T05:27:48.742564+0000","last_clean":"2024-09-17T05:27:48.742564+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:48.742564+0000","last_undegraded":"2024-09-17T05:27:48.742564+0000","last_fullsized":"2024-09-17T05:27:48.742564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488502+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:32.488502+0000","last_peered":"2024-09-17T05:27:32.488502+0000","last_clean":"2024-09-17T05:27:32.488502+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:32.488502+0000","last_undegraded":"2024-09-17T05:27:32.488502+0000","last_fullsized":"2024-09-17T05:27:32.488502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749241+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:48.749241+0000","last_peered":"2024-09-17T05:27:48.749241+0000","last_clean":"2024-09-17T05:27:48.749241+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:48.749241+0000","last_undegraded":"2024-09-17T05:27:48.749241+0000","last_fullsized":"2024-09-17T05:27:48.749241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746867+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:48.746867+0000","last_peered":"2024-09-17T05:27:48.746867+0000","last_clean":"2024-09-17T05:27:48.746867+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:48.746867+0000","last_undegraded":"2024-09-17T05:27:48.746867+0000","last_fullsized":"2024-09-17T05:27:48.746867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'11","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747548+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:48.747548+0000","last_peered":"2024-09-17T05:27:48.747548+0000","last_clean":"2024-09-17T05:27:48.747548+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:48.747548+0000","last_undegraded":"2024-09-17T05:27:48.747548+0000","last_fullsized":"2024-09-17T05:27:48.747548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488889+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488889+0000","last_peered":"2024-09-17T05:27:32.488889+0000","last_clean":"2024-09-17T05:27:32.488889+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488889+0000","last_undegraded":"2024-09-17T05:27:32.488889+0000","last_fullsized":"2024-09-17T05:27:32.488889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746302+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:48.746302+0000","last_peered":"2024-09-17T05:27:48.746302+0000","last_clean":"2024-09-17T05:27:48.746302+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:48.746302+0000","last_undegraded":"2024-09-17T05:27:48.746302+0000","last_fullsized":"2024-09-17T05:27:48.746302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747034+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:48.747034+0000","last_peered":"2024-09-17T05:27:48.747034+0000","last_clean":"2024-09-17T05:27:48.747034+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:48.747034+0000","last_undegraded":"2024-09-17T05:27:48.747034+0000","last_fullsized":"2024-09-17T05:27:48.747034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488921+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488921+0000","last_peered":"2024-09-17T05:27:32.488921+0000","last_clean":"2024-09-17T05:27:32.488921+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488921+0000","last_undegraded":"2024-09-17T05:27:32.488921+0000","last_fullsized":"2024-09-17T05:27:32.488921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686092+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:31.686092+0000","last_peered":"2024-09-17T05:27:31.686092+0000","last_clean":"2024-09-17T05:27:31.686092+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:31.686092+0000","last_undegraded":"2024-09-17T05:27:31.686092+0000","last_fullsized":"2024-09-17T05:27:31.686092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"24'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490498+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:32.490498+0000","last_peered":"2024-09-17T05:27:32.490498+0000","last_clean":"2024-09-17T05:27:32.490498+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:32.490498+0000","last_undegraded":"2024-09-17T05:27:32.490498+0000","last_fullsized":"2024-09-17T05:27:32.490498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687008+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:31.687008+0000","last_peered":"2024-09-17T05:27:31.687008+0000","last_clean":"2024-09-17T05:27:31.687008+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:31.687008+0000","last_undegraded":"2024-09-17T05:27:31.687008+0000","last_fullsized":"2024-09-17T05:27:31.687008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5424,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":11470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":102667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509451,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":692591,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":102685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":106577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":331776,"data_stored":692523,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:27:51.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:27:51.628 INFO:tasks.ceph.mds.f.smithi119.stderr:2024-09-17T05:27:51.626+0000 7f6f66926640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-17T05:27:51.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 -- 172.21.15.117:0/3266183245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe00b52b0 msgr2=0x7fdfe00b56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3266183245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe00b52b0 0x7fdfe00b56c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00ce430 tx=0x7fdfd002a5f0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 -- 172.21.15.117:0/3266183245 shutdown_connections 2024-09-17T05:27:51.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3266183245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe00b52b0 0x7fdfe00b56c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3266183245 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfe006dbe0 0x7fdfe006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3266183245 >> v2:172.21.15.119:3300/0 conn(0x7fdfe015bd70 0x7fdfe00ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 -- 172.21.15.117:0/3266183245 >> 172.21.15.117:0/3266183245 conn(0x7fdfe0074d80 msgr2=0x7fdfe00d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 -- 172.21.15.117:0/3266183245 shutdown_connections 2024-09-17T05:27:51.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.745+0000 7fdfe7157640 1 -- 172.21.15.117:0/3266183245 wait complete. 2024-09-17T05:27:51.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.747+0000 7fdfe7157640 1 Processor -- start 2024-09-17T05:27:51.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.747+0000 7fdfe7157640 1 -- start start 2024-09-17T05:27:51.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.748+0000 7fdfe7157640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 0x7fdfe01f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.748+0000 7fdfe7157640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfe00b52b0 0x7fdfe015db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.748+0000 7fdfe7157640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfe015bd70 0x7fdfe0152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.748+0000 7fdfe7157640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe00cacb0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.748+0000 7fdfe7157640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe00cd0d0 con 0x7fdfe015bd70 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.748+0000 7fdfe7157640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe01558b0 con 0x7fdfe00b52b0 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 0x7fdfe01f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 0x7fdfe01f4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40462/0 (socket says 172.21.15.117:40462) 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 -- 172.21.15.117:0/3977492592 learned_addr learned my addr 172.21.15.117:0/3977492592 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfd7fff640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfe00b52b0 0x7fdfe015db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe56f4640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfe015bd70 0x7fdfe0152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 -- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfe00b52b0 msgr2=0x7fdfe015db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfe00b52b0 0x7fdfe015db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 -- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfe015bd70 msgr2=0x7fdfe0152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfe015bd70 0x7fdfe0152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 -- 172.21.15.117:0/3977492592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe015d1e0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe4ef3640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 0x7fdfe01f4930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00e7c0 tx=0x7fdfdc00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:51.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc00cab0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfdc00cd40 con 0x7fdfe006dbe0 2024-09-17T05:27:51.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe015f4b0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc010690 con 0x7fdfe006dbe0 2024-09-17T05:27:51.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.749+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe015c9e0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.750+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe00cb290 con 0x7fdfe006dbe0 2024-09-17T05:27:51.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.751+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdfdc010830 con 0x7fdfe006dbe0 2024-09-17T05:27:51.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.752+0000 7fdfd57fa640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfcc078b90 0x7fdfcc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:51.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.752+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdfdc09c0b0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.752+0000 7fdfd7fff640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfcc078b90 0x7fdfcc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:51.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.752+0000 7fdfd7fff640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfcc078b90 0x7fdfcc07b0a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0033000 tx=0x7fdfd0004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:51.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.755+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdfdc063cd0 con 0x7fdfe006dbe0 2024-09-17T05:27:51.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.927+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdfe006dff0 con 0x7fdfcc078b90 2024-09-17T05:27:51.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.933+0000 7fdfd57fa640 1 -- 172.21.15.117:0/3977492592 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332736 (secure 0 0 0) 0x7fdfe006dff0 con 0x7fdfcc078b90 2024-09-17T05:27:51.938 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:51.940 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:27:51.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfcc078b90 msgr2=0x7fdfcc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfcc078b90 0x7fdfcc07b0a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0033000 tx=0x7fdfd0004730 comp rx=0 tx=0).stop 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 msgr2=0x7fdfe01f4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 0x7fdfe01f4930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00e7c0 tx=0x7fdfdc00ebc0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 shutdown_connections 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfcc078b90 0x7fdfcc07b0a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfe015bd70 0x7fdfe0152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfe00b52b0 0x7fdfe015db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 --2- 172.21.15.117:0/3977492592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe006dbe0 0x7fdfe01f4930 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 >> 172.21.15.117:0/3977492592 conn(0x7fdfe0074d80 msgr2=0x7fdfe0154f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 shutdown_connections 2024-09-17T05:27:51.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:51.936+0000 7fdfe7157640 1 -- 172.21.15.117:0/3977492592 wait complete. 2024-09-17T05:27:51.949 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-09-17T05:27:51.355357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595792,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":186,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218436,"kb_used_data":2232,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756284,"statfs":{"total":773094113280,"available":772870434816,"internally_reserved":0,"allocated":2285568,"data_stored":2007926,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3014,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004049"},"pg_stats":[{"pgid":"2.1f","version":"24'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688102+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:31.688102+0000","last_peered":"2024-09-17T05:27:31.688102+0000","last_clean":"2024-09-17T05:27:31.688102+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:31.688102+0000","last_undegraded":"2024-09-17T05:27:31.688102+0000","last_fullsized":"2024-09-17T05:27:31.688102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688831+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688831+0000","last_peered":"2024-09-17T05:27:31.688831+0000","last_clean":"2024-09-17T05:27:31.688831+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688831+0000","last_undegraded":"2024-09-17T05:27:31.688831+0000","last_fullsized":"2024-09-17T05:27:31.688831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490251+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:32.490251+0000","last_peered":"2024-09-17T05:27:32.490251+0000","last_clean":"2024-09-17T05:27:32.490251+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:32.490251+0000","last_undegraded":"2024-09-17T05:27:32.490251+0000","last_fullsized":"2024-09-17T05:27:32.490251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749303+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:48.749303+0000","last_peered":"2024-09-17T05:27:48.749303+0000","last_clean":"2024-09-17T05:27:48.749303+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:48.749303+0000","last_undegraded":"2024-09-17T05:27:48.749303+0000","last_fullsized":"2024-09-17T05:27:48.749303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686433+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686433+0000","last_peered":"2024-09-17T05:27:31.686433+0000","last_clean":"2024-09-17T05:27:31.686433+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686433+0000","last_undegraded":"2024-09-17T05:27:31.686433+0000","last_fullsized":"2024-09-17T05:27:31.686433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747002+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:48.747002+0000","last_peered":"2024-09-17T05:27:48.747002+0000","last_clean":"2024-09-17T05:27:48.747002+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:48.747002+0000","last_undegraded":"2024-09-17T05:27:48.747002+0000","last_fullsized":"2024-09-17T05:27:48.747002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.748010+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:48.748010+0000","last_peered":"2024-09-17T05:27:48.748010+0000","last_clean":"2024-09-17T05:27:48.748010+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:48.748010+0000","last_undegraded":"2024-09-17T05:27:48.748010+0000","last_fullsized":"2024-09-17T05:27:48.748010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750067+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:48.750067+0000","last_peered":"2024-09-17T05:27:48.750067+0000","last_clean":"2024-09-17T05:27:48.750067+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:48.750067+0000","last_undegraded":"2024-09-17T05:27:48.750067+0000","last_fullsized":"2024-09-17T05:27:48.750067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686707+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:31.686707+0000","last_peered":"2024-09-17T05:27:31.686707+0000","last_clean":"2024-09-17T05:27:31.686707+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:31.686707+0000","last_undegraded":"2024-09-17T05:27:31.686707+0000","last_fullsized":"2024-09-17T05:27:31.686707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490238+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490238+0000","last_peered":"2024-09-17T05:27:32.490238+0000","last_clean":"2024-09-17T05:27:32.490238+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490238+0000","last_undegraded":"2024-09-17T05:27:32.490238+0000","last_fullsized":"2024-09-17T05:27:32.490238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490613+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490613+0000","last_peered":"2024-09-17T05:27:32.490613+0000","last_clean":"2024-09-17T05:27:32.490613+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490613+0000","last_undegraded":"2024-09-17T05:27:32.490613+0000","last_fullsized":"2024-09-17T05:27:32.490613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"25'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488421+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:32.488421+0000","last_peered":"2024-09-17T05:27:32.488421+0000","last_clean":"2024-09-17T05:27:32.488421+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:32.488421+0000","last_undegraded":"2024-09-17T05:27:32.488421+0000","last_fullsized":"2024-09-17T05:27:32.488421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489023+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:32.489023+0000","last_peered":"2024-09-17T05:27:32.489023+0000","last_clean":"2024-09-17T05:27:32.489023+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:32.489023+0000","last_undegraded":"2024-09-17T05:27:32.489023+0000","last_fullsized":"2024-09-17T05:27:32.489023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"25'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749871+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:48.749871+0000","last_peered":"2024-09-17T05:27:48.749871+0000","last_clean":"2024-09-17T05:27:48.749871+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:48.749871+0000","last_undegraded":"2024-09-17T05:27:48.749871+0000","last_fullsized":"2024-09-17T05:27:48.749871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687718+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:31.687718+0000","last_peered":"2024-09-17T05:27:31.687718+0000","last_clean":"2024-09-17T05:27:31.687718+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:31.687718+0000","last_undegraded":"2024-09-17T05:27:31.687718+0000","last_fullsized":"2024-09-17T05:27:31.687718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489640+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489640+0000","last_peered":"2024-09-17T05:27:32.489640+0000","last_clean":"2024-09-17T05:27:32.489640+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489640+0000","last_undegraded":"2024-09-17T05:27:32.489640+0000","last_fullsized":"2024-09-17T05:27:32.489640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"25'10","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.742564+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:48.742564+0000","last_peered":"2024-09-17T05:27:48.742564+0000","last_clean":"2024-09-17T05:27:48.742564+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:48.742564+0000","last_undegraded":"2024-09-17T05:27:48.742564+0000","last_fullsized":"2024-09-17T05:27:48.742564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488502+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:32.488502+0000","last_peered":"2024-09-17T05:27:32.488502+0000","last_clean":"2024-09-17T05:27:32.488502+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:32.488502+0000","last_undegraded":"2024-09-17T05:27:32.488502+0000","last_fullsized":"2024-09-17T05:27:32.488502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749241+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:48.749241+0000","last_peered":"2024-09-17T05:27:48.749241+0000","last_clean":"2024-09-17T05:27:48.749241+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:48.749241+0000","last_undegraded":"2024-09-17T05:27:48.749241+0000","last_fullsized":"2024-09-17T05:27:48.749241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746867+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:48.746867+0000","last_peered":"2024-09-17T05:27:48.746867+0000","last_clean":"2024-09-17T05:27:48.746867+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:48.746867+0000","last_undegraded":"2024-09-17T05:27:48.746867+0000","last_fullsized":"2024-09-17T05:27:48.746867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'11","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747548+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:48.747548+0000","last_peered":"2024-09-17T05:27:48.747548+0000","last_clean":"2024-09-17T05:27:48.747548+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:48.747548+0000","last_undegraded":"2024-09-17T05:27:48.747548+0000","last_fullsized":"2024-09-17T05:27:48.747548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488889+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488889+0000","last_peered":"2024-09-17T05:27:32.488889+0000","last_clean":"2024-09-17T05:27:32.488889+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488889+0000","last_undegraded":"2024-09-17T05:27:32.488889+0000","last_fullsized":"2024-09-17T05:27:32.488889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746302+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:48.746302+0000","last_peered":"2024-09-17T05:27:48.746302+0000","last_clean":"2024-09-17T05:27:48.746302+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:48.746302+0000","last_undegraded":"2024-09-17T05:27:48.746302+0000","last_fullsized":"2024-09-17T05:27:48.746302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747034+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:48.747034+0000","last_peered":"2024-09-17T05:27:48.747034+0000","last_clean":"2024-09-17T05:27:48.747034+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:48.747034+0000","last_undegraded":"2024-09-17T05:27:48.747034+0000","last_fullsized":"2024-09-17T05:27:48.747034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488921+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488921+0000","last_peered":"2024-09-17T05:27:32.488921+0000","last_clean":"2024-09-17T05:27:32.488921+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488921+0000","last_undegraded":"2024-09-17T05:27:32.488921+0000","last_fullsized":"2024-09-17T05:27:32.488921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686092+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:31.686092+0000","last_peered":"2024-09-17T05:27:31.686092+0000","last_clean":"2024-09-17T05:27:31.686092+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:31.686092+0000","last_undegraded":"2024-09-17T05:27:31.686092+0000","last_fullsized":"2024-09-17T05:27:31.686092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"24'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490498+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:32.490498+0000","last_peered":"2024-09-17T05:27:32.490498+0000","last_clean":"2024-09-17T05:27:32.490498+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:32.490498+0000","last_undegraded":"2024-09-17T05:27:32.490498+0000","last_fullsized":"2024-09-17T05:27:32.490498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687008+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:31.687008+0000","last_peered":"2024-09-17T05:27:31.687008+0000","last_clean":"2024-09-17T05:27:31.687008+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:31.687008+0000","last_undegraded":"2024-09-17T05:27:31.687008+0000","last_fullsized":"2024-09-17T05:27:31.687008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5424,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":11470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":102667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509451,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":105905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":692591,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":102685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":106577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":331776,"data_stored":692523,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:27:51.954 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T05:27:51.954 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T05:27:51.954 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T05:27:51.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501e45c640 1 -- 172.21.15.117:0/982053849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5000004d40 msgr2=0x7f5000005170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501e45c640 1 --2- 172.21.15.117:0/982053849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5000004d40 0x7f5000005170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f501815b730 tx=0x7f5000005830 comp rx=0 tx=0).stop 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501c9f9640 1 -- 172.21.15.117:0/982053849 reap_dead start 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501e45c640 1 -- 172.21.15.117:0/982053849 shutdown_connections 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501e45c640 1 -- 172.21.15.117:0/982053849 >> 172.21.15.117:0/982053849 conn(0x7f5018075e00 msgr2=0x7f501806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501e45c640 1 -- 172.21.15.117:0/982053849 shutdown_connections 2024-09-17T05:27:52.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.100+0000 7f501e45c640 1 -- 172.21.15.117:0/982053849 wait complete. 2024-09-17T05:27:52.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.102+0000 7f501e45c640 1 Processor -- start 2024-09-17T05:27:52.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.102+0000 7f501e45c640 1 -- start start 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.103+0000 7f501e45c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5018155390 0x7f50181e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.103+0000 7f501e45c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 0x7f50181f4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.103+0000 7f501e45c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50180cd980 0x7f5018157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.103+0000 7f501e45c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f50180c5da0 con 0x7f50180cd570 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.103+0000 7f501e45c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f501815f6b0 con 0x7f50180cd980 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.103+0000 7f501e45c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f50181558b0 con 0x7f5018155390 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 0x7f50181f4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 0x7f50181f4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40482/0 (socket says 172.21.15.117:40482) 2024-09-17T05:27:52.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 -- 172.21.15.117:0/876265669 learned_addr learned my addr 172.21.15.117:0/876265669 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f5017fff640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5018155390 0x7f50181e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f501c9f9640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50180cd980 0x7f5018157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 -- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5018155390 msgr2=0x7f50181e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5018155390 0x7f50181e57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 -- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50180cd980 msgr2=0x7f5018157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50180cd980 0x7f5018157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 -- 172.21.15.117:0/876265669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50181e5e00 con 0x7f50180cd570 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f50177fe640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 0x7f50181f4b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f500c006400 tx=0x7f500c003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c015070 con 0x7f50180cd570 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f500c011940 con 0x7f50180cd570 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c011d90 con 0x7f50180cd570 2024-09-17T05:27:52.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50181f1760 con 0x7f50180cd570 2024-09-17T05:27:52.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.104+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50181f1d20 con 0x7f50180cd570 2024-09-17T05:27:52.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.105+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50180c79e0 con 0x7f50180cd570 2024-09-17T05:27:52.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.106+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f500c018070 con 0x7f50180cd570 2024-09-17T05:27:52.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.107+0000 7f5014ff9640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5004078bc0 0x7f500407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.107+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f500c09bd00 con 0x7f50180cd570 2024-09-17T05:27:52.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.107+0000 7f5017fff640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5004078bc0 0x7f500407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.107+0000 7f5017fff640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5004078bc0 0x7f500407b0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f50080036b0 tx=0x7f5008006110 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:52.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.111+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f500c0637b0 con 0x7f50180cd570 2024-09-17T05:27:52.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:27:52.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:27:52.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.325+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f501815c180 con 0x7f50180cd570 2024-09-17T05:27:52.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.326+0000 7f5014ff9640 1 -- 172.21.15.117:0/876265669 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f500c0681e0 con 0x7f50180cd570 2024-09-17T05:27:52.328 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:52.328 INFO:teuthology.orchestra.run.smithi117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5004078bc0 msgr2=0x7f500407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5004078bc0 0x7f500407b0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f50080036b0 tx=0x7f5008006110 comp rx=0 tx=0).stop 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 msgr2=0x7f50181f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 0x7f50181f4b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f500c006400 tx=0x7f500c003aa0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 shutdown_connections 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5004078bc0 0x7f500407b0d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50180cd980 0x7f5018157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50180cd570 0x7f50181f4b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 --2- 172.21.15.117:0/876265669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5018155390 0x7f50181e57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 >> 172.21.15.117:0/876265669 conn(0x7f5018075e00 msgr2=0x7f5018158f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 shutdown_connections 2024-09-17T05:27:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.328+0000 7f501e45c640 1 -- 172.21.15.117:0/876265669 wait complete. 2024-09-17T05:27:52.341 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T05:27:52.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 -- 172.21.15.117:0/2176976852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 msgr2=0x7f590406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 --2- 172.21.15.117:0/2176976852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f590406dff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f59040ce430 tx=0x7f58f402a5f0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 -- 172.21.15.117:0/2176976852 shutdown_connections 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 --2- 172.21.15.117:0/2176976852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f590406dff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 --2- 172.21.15.117:0/2176976852 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 0x7f59040b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 --2- 172.21.15.117:0/2176976852 >> v2:172.21.15.119:3300/0 conn(0x7f590415bd70 0x7f59040ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 -- 172.21.15.117:0/2176976852 >> 172.21.15.117:0/2176976852 conn(0x7f5904074d80 msgr2=0x7f59040d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 -- 172.21.15.117:0/2176976852 shutdown_connections 2024-09-17T05:27:52.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.451+0000 7f59092d5640 1 -- 172.21.15.117:0/2176976852 wait complete. 2024-09-17T05:27:52.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.453+0000 7f59092d5640 1 Processor -- start 2024-09-17T05:27:52.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.453+0000 7f59092d5640 1 -- start start 2024-09-17T05:27:52.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f59092d5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f59041f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f59092d5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 0x7f590415ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f59092d5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f590415bd70 0x7f590415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f59092d5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f59040cacb0 con 0x7f590406dbe0 2024-09-17T05:27:52.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f59092d5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f59040cd0d0 con 0x7f590415bd70 2024-09-17T05:27:52.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f59092d5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f59041558b0 con 0x7f59040b52b0 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f5902ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f59041f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f5902ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f59041f51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40494/0 (socket says 172.21.15.117:40494) 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.454+0000 7f5902ffd640 1 -- 172.21.15.117:0/714058354 learned_addr learned my addr 172.21.15.117:0/714058354 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f59027fc640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 0x7f590415ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f59037fe640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f590415bd70 0x7f590415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f5902ffd640 1 -- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 msgr2=0x7f590415ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f5902ffd640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 0x7f590415ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f5902ffd640 1 -- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f590415bd70 msgr2=0x7f590415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f5902ffd640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f590415bd70 0x7f590415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f5902ffd640 1 -- 172.21.15.117:0/714058354 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59041f5720 con 0x7f590406dbe0 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f59037fe640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f590415bd70 0x7f590415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:52.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f59027fc640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 0x7f590415ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:52.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f5902ffd640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f59041f51e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00eb60 tx=0x7f58fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:52.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc010630 con 0x7f590406dbe0 2024-09-17T05:27:52.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58fc0108c0 con 0x7f590406dbe0 2024-09-17T05:27:52.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59040c69c0 con 0x7f590406dbe0 2024-09-17T05:27:52.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc010d10 con 0x7f590406dbe0 2024-09-17T05:27:52.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.455+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59040c6f80 con 0x7f590406dbe0 2024-09-17T05:27:52.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.456+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59040cb290 con 0x7f590406dbe0 2024-09-17T05:27:52.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.457+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f58fc018420 con 0x7f590406dbe0 2024-09-17T05:27:52.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.457+0000 7f58fb7fe640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f58e4078bc0 0x7f58e407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.458+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f58fc09bdd0 con 0x7f590406dbe0 2024-09-17T05:27:52.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.458+0000 7f59027fc640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f58e4078bc0 0x7f58e407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.458+0000 7f59027fc640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f58e4078bc0 0x7f58e407b0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59040c8530 tx=0x7f58ec005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:52.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.462+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f58fc00ee60 con 0x7f590406dbe0 2024-09-17T05:27:52.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.671+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59040ce920 con 0x7f590406dbe0 2024-09-17T05:27:52.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.672+0000 7f58fb7fe640 1 -- 172.21.15.117:0/714058354 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6918 (secure 0 0 0) 0x7f58fc09d090 con 0x7f590406dbe0 2024-09-17T05:27:52.673 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:52.673 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"btime":"2024-09-17T05:27:49:758599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:49.758588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:27:52.674 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 13 2024-09-17T05:27:52.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f58e4078bc0 msgr2=0x7f58e407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f58e4078bc0 0x7f58e407b0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59040c8530 tx=0x7f58ec005f90 comp rx=0 tx=0).stop 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 msgr2=0x7f59041f51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f59041f51e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00eb60 tx=0x7f58fc00c6a0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 shutdown_connections 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f58e4078bc0 0x7f58e407b0d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f590415bd70 0x7f590415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f59040b52b0 0x7f590415ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 --2- 172.21.15.117:0/714058354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f590406dbe0 0x7f59041f51e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 >> 172.21.15.117:0/714058354 conn(0x7f5904074d80 msgr2=0x7f5904159160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:52.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 shutdown_connections 2024-09-17T05:27:52.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.674+0000 7f59092d5640 1 -- 172.21.15.117:0/714058354 wait complete. 2024-09-17T05:27:52.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:52.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 -- 172.21.15.117:0/2652125182 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 msgr2=0x7f2654165c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2652125182 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 0x7f2654165c70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f264c005fd0 tx=0x7f264c02a670 comp rx=0 tx=0).stop 2024-09-17T05:27:52.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 -- 172.21.15.117:0/2652125182 shutdown_connections 2024-09-17T05:27:52.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2652125182 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 0x7f2654165c70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2652125182 >> v2:172.21.15.119:3300/0 conn(0x7f265415b2f0 0x7f26540c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 -- 172.21.15.117:0/2652125182 >> 172.21.15.117:0/2652125182 conn(0x7f2654075e00 msgr2=0x7f265406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:52.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 -- 172.21.15.117:0/2652125182 shutdown_connections 2024-09-17T05:27:52.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.798+0000 7f265ad8b640 1 -- 172.21.15.117:0/2652125182 wait complete. 2024-09-17T05:27:52.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.800+0000 7f265ad8b640 1 Processor -- start 2024-09-17T05:27:52.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.800+0000 7f265ad8b640 1 -- start start 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f265ad8b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 0x7f26540c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f265ad8b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 0x7f2654155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f265ad8b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f265415b2f0 0x7f2654151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f265ad8b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f26540c5da0 con 0x7f26540cb390 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f265ad8b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f265415f6b0 con 0x7f265415b2f0 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f265ad8b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f26540c6c10 con 0x7f26540c6800 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.801+0000 7f2653fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 0x7f2654155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 0x7f2654155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40496/0 (socket says 172.21.15.117:40496) 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 -- 172.21.15.117:0/2963216418 learned_addr learned my addr 172.21.15.117:0/2963216418 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2659328640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f265415b2f0 0x7f2654151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2658b27640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 0x7f26540c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 -- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 msgr2=0x7f26540c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 0x7f26540c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 -- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f265415b2f0 msgr2=0x7f2654151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f265415b2f0 0x7f2654151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 -- 172.21.15.117:0/2963216418 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26541584c0 con 0x7f26540cb390 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f2653fff640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 0x7f2654155e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f264800ea20 tx=0x7f264800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264800cd80 con 0x7f26540cb390 2024-09-17T05:27:52.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2648010430 con 0x7f26540cb390 2024-09-17T05:27:52.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26541e6970 con 0x7f26540cb390 2024-09-17T05:27:52.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648010860 con 0x7f26540cb390 2024-09-17T05:27:52.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.802+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2654157cc0 con 0x7f26540cb390 2024-09-17T05:27:52.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.803+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26540c7430 con 0x7f26540cb390 2024-09-17T05:27:52.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.804+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2648010a00 con 0x7f26540cb390 2024-09-17T05:27:52.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.805+0000 7f26517fa640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2640078b90 0x7f264007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:52.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.805+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f264809ba20 con 0x7f26540cb390 2024-09-17T05:27:52.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.805+0000 7f2658b27640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2640078b90 0x7f264007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:52.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.805+0000 7f2658b27640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2640078b90 0x7f264007b0a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2644001e20 tx=0x7f2644009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:52.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:52.809+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f26540c7430 con 0x7f26540cb390 2024-09-17T05:27:53.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.017+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26540c7950 con 0x7f26540cb390 2024-09-17T05:27:53.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.018+0000 7f26517fa640 1 -- 172.21.15.117:0/2963216418 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6918 (secure 0 0 0) 0x7f26480634e0 con 0x7f26540cb390 2024-09-17T05:27:53.020 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:53.020 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"btime":"2024-09-17T05:27:49:758599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:49.758588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:27:53.020 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 13 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2640078b90 msgr2=0x7f264007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2640078b90 0x7f264007b0a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2644001e20 tx=0x7f2644009040 comp rx=0 tx=0).stop 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 msgr2=0x7f2654155e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 0x7f2654155e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f264800ea20 tx=0x7f264800eee0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 shutdown_connections 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2640078b90 0x7f264007b0a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f265415b2f0 0x7f2654151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26540cb390 0x7f2654155e20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 --2- 172.21.15.117:0/2963216418 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f26540c6800 0x7f26540c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 >> 172.21.15.117:0/2963216418 conn(0x7f2654075e00 msgr2=0x7f26540cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 shutdown_connections 2024-09-17T05:27:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.020+0000 7f265ad8b640 1 -- 172.21.15.117:0/2963216418 wait complete. 2024-09-17T05:27:53.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:27:53.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 -- 172.21.15.117:0/1205862695 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 msgr2=0x7f11000c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 --2- 172.21.15.117:0/1205862695 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 0x7f11000c4f50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009860 tx=0x7f10f002a510 comp rx=0 tx=0).stop 2024-09-17T05:27:53.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 -- 172.21.15.117:0/1205862695 shutdown_connections 2024-09-17T05:27:53.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 --2- 172.21.15.117:0/1205862695 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 0x7f11000c4f50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 -- 172.21.15.117:0/1205862695 >> 172.21.15.117:0/1205862695 conn(0x7f110006e590 msgr2=0x7f11000cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:53.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 -- 172.21.15.117:0/1205862695 shutdown_connections 2024-09-17T05:27:53.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.143+0000 7f1107a30640 1 -- 172.21.15.117:0/1205862695 wait complete. 2024-09-17T05:27:53.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.145+0000 7f1107a30640 1 Processor -- start 2024-09-17T05:27:53.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.145+0000 7f1107a30640 1 -- start start 2024-09-17T05:27:53.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f1107a30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 0x7f1100148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f1107a30640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11000c67a0 0x7f11001f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f1107a30640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 0x7f110013f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f1107a30640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11000ce0d0 con 0x7f11000c5490 2024-09-17T05:27:53.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f1107a30640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f11000c58a0 con 0x7f11000c67a0 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f1107a30640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f11000c7ff0 con 0x7f1100152210 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f11057cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 0x7f1100148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f11057cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 0x7f1100148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40506/0 (socket says 172.21.15.117:40506) 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.146+0000 7f11057cc640 1 -- 172.21.15.117:0/1957710777 learned_addr learned my addr 172.21.15.117:0/1957710777 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f1104fcb640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11000c67a0 0x7f11001f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f1105fcd640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 0x7f110013f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f11057cc640 1 -- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 msgr2=0x7f110013f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f11057cc640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 0x7f110013f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f11057cc640 1 -- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11000c67a0 msgr2=0x7f11001f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f11057cc640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11000c67a0 0x7f11001f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f11057cc640 1 -- 172.21.15.117:0/1957710777 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11001f3ad0 con 0x7f11000c5490 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f11057cc640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 0x7f1100148180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00c460 tx=0x7f10fc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:53.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc016020 con 0x7f11000c5490 2024-09-17T05:27:53.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10fc012a00 con 0x7f11000c5490 2024-09-17T05:27:53.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11001469d0 con 0x7f11000c5490 2024-09-17T05:27:53.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc012e50 con 0x7f11000c5490 2024-09-17T05:27:53.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.147+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11001f3070 con 0x7f11000c5490 2024-09-17T05:27:53.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.148+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1100146ba0 con 0x7f11000c5490 2024-09-17T05:27:53.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.149+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10fc019020 con 0x7f11000c5490 2024-09-17T05:27:53.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.149+0000 7f10f5ffb640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f10e4078d60 0x7f10e407b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.149+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f10fc09ab20 con 0x7f11000c5490 2024-09-17T05:27:53.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.150+0000 7f1104fcb640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f10e4078d60 0x7f10e407b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.150+0000 7f1104fcb640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f10e4078d60 0x7f10e407b270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10f0034000 tx=0x7f10f0004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:53.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.154+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f10fc020030 con 0x7f11000c5490 2024-09-17T05:27:53.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.328+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11000c4fc0 con 0x7f11000c5490 2024-09-17T05:27:53.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.329+0000 7f10f5ffb640 1 -- 172.21.15.117:0/1957710777 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f10fc0626f0 con 0x7f11000c5490 2024-09-17T05:27:53.331 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:53.331 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:27:53.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f10e4078d60 msgr2=0x7f10e407b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f10e4078d60 0x7f10e407b270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10f0034000 tx=0x7f10f0004730 comp rx=0 tx=0).stop 2024-09-17T05:27:53.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 msgr2=0x7f1100148180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 0x7f1100148180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00c460 tx=0x7f10fc00c920 comp rx=0 tx=0).stop 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 shutdown_connections 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f10e4078d60 0x7f10e407b270 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1100152210 0x7f110013f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11000c67a0 0x7f11001f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 --2- 172.21.15.117:0/1957710777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11000c5490 0x7f1100148180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 >> 172.21.15.117:0/1957710777 conn(0x7f110006e590 msgr2=0x7f1100142740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 shutdown_connections 2024-09-17T05:27:53.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.332+0000 7f1107a30640 1 -- 172.21.15.117:0/1957710777 wait complete. 2024-09-17T05:27:53.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 -- 172.21.15.117:0/2493212693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e340747b0 msgr2=0x7f6e341521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 --2- 172.21.15.117:0/2493212693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e340747b0 0x7f6e341521e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6e340c3180 tx=0x7f6e2402a5f0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 -- 172.21.15.117:0/2493212693 shutdown_connections 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 --2- 172.21.15.117:0/2493212693 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e3406fe20 0x7f6e34070230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 --2- 172.21.15.117:0/2493212693 >> v2:172.21.15.119:3300/0 conn(0x7f6e3415be40 0x7f6e34074bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 --2- 172.21.15.117:0/2493212693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e340747b0 0x7f6e341521e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 -- 172.21.15.117:0/2493212693 >> 172.21.15.117:0/2493212693 conn(0x7f6e3406cda0 msgr2=0x7f6e3406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 -- 172.21.15.117:0/2493212693 shutdown_connections 2024-09-17T05:27:53.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.456+0000 7f6e39207640 1 -- 172.21.15.117:0/2493212693 wait complete. 2024-09-17T05:27:53.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.458+0000 7f6e39207640 1 Processor -- start 2024-09-17T05:27:53.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.458+0000 7f6e39207640 1 -- start start 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e39207640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e3406fe20 0x7f6e341537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e39207640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 0x7f6e340c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e39207640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e34145a40 0x7f6e340c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e39207640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3415c250 con 0x7f6e3415be40 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e39207640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6e340c3ab0 con 0x7f6e3406fe20 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e39207640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6e3415c3d0 con 0x7f6e34145a40 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 0x7f6e340c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 0x7f6e340c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40522/0 (socket says 172.21.15.117:40522) 2024-09-17T05:27:53.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 -- 172.21.15.117:0/3996320248 learned_addr learned my addr 172.21.15.117:0/3996320248 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e32d9d640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e3406fe20 0x7f6e341537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e3359e640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e34145a40 0x7f6e340c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 -- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e34145a40 msgr2=0x7f6e340c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e34145a40 0x7f6e340c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 -- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e3406fe20 msgr2=0x7f6e341537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e3406fe20 0x7f6e341537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.459+0000 7f6e2bfff640 1 -- 172.21.15.117:0/3996320248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e34152b90 con 0x7f6e3415be40 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.460+0000 7f6e2bfff640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 0x7f6e340c76f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00d770 tx=0x7f6e1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.460+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c017070 con 0x7f6e3415be40 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.460+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e1c004310 con 0x7f6e3415be40 2024-09-17T05:27:53.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.460+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e340c7c30 con 0x7f6e3415be40 2024-09-17T05:27:53.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.460+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c009c30 con 0x7f6e3415be40 2024-09-17T05:27:53.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.460+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e340c9530 con 0x7f6e3415be40 2024-09-17T05:27:53.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.461+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e340735a0 con 0x7f6e3415be40 2024-09-17T05:27:53.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.461+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6e1c005020 con 0x7f6e3415be40 2024-09-17T05:27:53.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.462+0000 7f6e2b7fe640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e14078b90 0x7f6e1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:53.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.462+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6e1c09b180 con 0x7f6e3415be40 2024-09-17T05:27:53.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.462+0000 7f6e32d9d640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e14078b90 0x7f6e1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:53.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.463+0000 7f6e32d9d640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e14078b90 0x7f6e1407b0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6e24033000 tx=0x7f6e24004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:53.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.467+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6e340735a0 con 0x7f6e3415be40 2024-09-17T05:27:53.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.678+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e340cc680 con 0x7f6e3415be40 2024-09-17T05:27:53.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.679+0000 7f6e2b7fe640 1 -- 172.21.15.117:0/3996320248 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6918 (secure 0 0 0) 0x7f6e340cc680 con 0x7f6e3415be40 2024-09-17T05:27:53.680 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:53.680 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"btime":"2024-09-17T05:27:49:758599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:49.758588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:27:53.680 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 13 2024-09-17T05:27:53.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e14078b90 msgr2=0x7f6e1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e14078b90 0x7f6e1407b0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6e24033000 tx=0x7f6e24004730 comp rx=0 tx=0).stop 2024-09-17T05:27:53.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 msgr2=0x7f6e340c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:53.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 0x7f6e340c76f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00d770 tx=0x7f6e1c00dc30 comp rx=0 tx=0).stop 2024-09-17T05:27:53.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 shutdown_connections 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e14078b90 0x7f6e1407b0a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e34145a40 0x7f6e340c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e3415be40 0x7f6e340c76f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 --2- 172.21.15.117:0/3996320248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e3406fe20 0x7f6e341537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 >> 172.21.15.117:0/3996320248 conn(0x7f6e3406cda0 msgr2=0x7f6e340cb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 shutdown_connections 2024-09-17T05:27:53.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:53.681+0000 7f6e39207640 1 -- 172.21.15.117:0/3996320248 wait complete. 2024-09-17T05:27:54.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T05:27:26.003433+0000', 'modified': '2024-09-17T05:27:49.758588+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4440, 'mds_1': 4434, 'mds_2': 4422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4422': {'gid': 4422, 'name': 'e', 'rank': 2, 'incarnation': 13, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.117:6837/242246295', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 242246295}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 242246295}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4434': {'gid': 4434, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.119:6835/2224210538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2224210538}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2224210538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4440': {'gid': 4440, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6839/2500745218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2500745218}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2500745218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4440, 'qdb_cluster': [4440, 4434]} 2024-09-17T05:27:54.299 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4422:up:creating 2024-09-17T05:27:55.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:55.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 -- 172.21.15.117:0/1688538426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 msgr2=0x7f6e800c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:55.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1688538426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e800c37c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e78005fd0 tx=0x7f6e7802a4f0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 -- 172.21.15.117:0/1688538426 shutdown_connections 2024-09-17T05:27:55.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1688538426 >> v2:172.21.15.119:3300/0 conn(0x7f6e80159b30 0x7f6e800c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1688538426 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e80153260 0x7f6e8015d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1688538426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e800c37c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 -- 172.21.15.117:0/1688538426 >> 172.21.15.117:0/1688538426 conn(0x7f6e80076430 msgr2=0x7f6e80072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:55.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 -- 172.21.15.117:0/1688538426 shutdown_connections 2024-09-17T05:27:55.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.426+0000 7f6e87b82640 1 -- 172.21.15.117:0/1688538426 wait complete. 2024-09-17T05:27:55.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.428+0000 7f6e87b82640 1 Processor -- start 2024-09-17T05:27:55.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.428+0000 7f6e87b82640 1 -- start start 2024-09-17T05:27:55.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e87b82640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e801e2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:55.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e87b82640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e80153260 0x7f6e801480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:55.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e87b82640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e80159b30 0x7f6e801e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:55.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e87b82640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80159f40 con 0x7f6e800cad30 2024-09-17T05:27:55.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e87b82640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6e800cbc30 con 0x7f6e80159b30 2024-09-17T05:27:55.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e87b82640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6e801e38a0 con 0x7f6e80153260 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e801e2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e801e2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42664/0 (socket says 172.21.15.117:42664) 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 -- 172.21.15.117:0/1895782122 learned_addr learned my addr 172.21.15.117:0/1895782122 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8511d640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e80153260 0x7f6e801480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8611f640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e80159b30 0x7f6e801e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 -- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e80153260 msgr2=0x7f6e801480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e80153260 0x7f6e801480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 -- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e80159b30 msgr2=0x7f6e801e3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e80159b30 0x7f6e801e3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 -- 172.21.15.117:0/1895782122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e801e33d0 con 0x7f6e800cad30 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.429+0000 7f6e8591e640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e801e2e90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006010 tx=0x7f6e780038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.430+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7803e030 con 0x7f6e800cad30 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.430+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e78003d90 con 0x7f6e800cad30 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.430+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78035040 con 0x7f6e800cad30 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.430+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e801f5440 con 0x7f6e800cad30 2024-09-17T05:27:55.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.430+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e801f5a00 con 0x7f6e800cad30 2024-09-17T05:27:55.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.431+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e800ca900 con 0x7f6e800cad30 2024-09-17T05:27:55.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.432+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6e78041030 con 0x7f6e800cad30 2024-09-17T05:27:55.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.432+0000 7f6e7e7fc640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e6c078b90 0x7f6e6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:55.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.432+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6e780bb7b0 con 0x7f6e800cad30 2024-09-17T05:27:55.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.432+0000 7f6e8511d640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e6c078b90 0x7f6e6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:55.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.433+0000 7f6e8511d640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e6c078b90 0x7f6e6c07b0a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6e800b4670 tx=0x7f6e7400b260 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:55.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.437+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6e78083260 con 0x7f6e800cad30 2024-09-17T05:27:55.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.648+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e800ca300 con 0x7f6e800cad30 2024-09-17T05:27:55.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.649+0000 7f6e7e7fc640 1 -- 172.21.15.117:0/1895782122 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6918 (secure 0 0 0) 0x7f6e78087c90 con 0x7f6e800cad30 2024-09-17T05:27:55.650 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:55.650 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"btime":"2024-09-17T05:27:49:758599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:49.758588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:27:55.650 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 13 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e6c078b90 msgr2=0x7f6e6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e6c078b90 0x7f6e6c07b0a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6e800b4670 tx=0x7f6e7400b260 comp rx=0 tx=0).stop 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 msgr2=0x7f6e801e2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e801e2e90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006010 tx=0x7f6e780038a0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 shutdown_connections 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6e6c078b90 0x7f6e6c07b0a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e80159b30 0x7f6e801e3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6e80153260 0x7f6e801480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 --2- 172.21.15.117:0/1895782122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e800cad30 0x7f6e801e2e90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:55.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 >> 172.21.15.117:0/1895782122 conn(0x7f6e80076430 msgr2=0x7f6e80072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:55.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 shutdown_connections 2024-09-17T05:27:55.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:55.651+0000 7f6e87b82640 1 -- 172.21.15.117:0/1895782122 wait complete. 2024-09-17T05:27:56.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T05:27:26.003433+0000', 'modified': '2024-09-17T05:27:49.758588+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4440, 'mds_1': 4434, 'mds_2': 4422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4422': {'gid': 4422, 'name': 'e', 'rank': 2, 'incarnation': 13, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.117:6837/242246295', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 242246295}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 242246295}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4434': {'gid': 4434, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.119:6835/2224210538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2224210538}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2224210538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4440': {'gid': 4440, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6839/2500745218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2500745218}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2500745218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4440, 'qdb_cluster': [4440, 4434]} 2024-09-17T05:27:56.269 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4422:up:creating 2024-09-17T05:27:56.627 INFO:tasks.ceph.mds.f.smithi119.stderr:2024-09-17T05:27:56.627+0000 7f6f66926640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-17T05:27:57.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:27:57.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.395+0000 7fbab3f50640 1 -- 172.21.15.117:0/706845957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac15f650 msgr2=0x7fbaac0cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:57.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.395+0000 7fbab3f50640 1 --2- 172.21.15.117:0/706845957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac15f650 0x7fbaac0cb230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0cc650 tx=0x7fbaa402aa30 comp rx=0 tx=0).stop 2024-09-17T05:27:57.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.395+0000 7fbab3f50640 1 -- 172.21.15.117:0/706845957 shutdown_connections 2024-09-17T05:27:57.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.395+0000 7fbab3f50640 1 --2- 172.21.15.117:0/706845957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac15f650 0x7fbaac0cb230 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.395+0000 7fbab3f50640 1 --2- 172.21.15.117:0/706845957 >> v2:172.21.15.119:3300/0 conn(0x7fbaac155860 0x7fbaac0c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.395+0000 7fbab3f50640 1 -- 172.21.15.117:0/706845957 >> 172.21.15.117:0/706845957 conn(0x7fbaac076430 msgr2=0x7fbaac072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:57.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.396+0000 7fbab3f50640 1 -- 172.21.15.117:0/706845957 shutdown_connections 2024-09-17T05:27:57.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.396+0000 7fbab3f50640 1 -- 172.21.15.117:0/706845957 wait complete. 2024-09-17T05:27:57.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.397+0000 7fbab3f50640 1 Processor -- start 2024-09-17T05:27:57.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.397+0000 7fbab3f50640 1 -- start start 2024-09-17T05:27:57.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.398+0000 7fbab3f50640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaac0c9c70 0x7fbaac158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:57.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.398+0000 7fbab3f50640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 0x7fbaac0c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:57.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.398+0000 7fbab3f50640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbaac15f650 0x7fbaac14f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.398+0000 7fbab3f50640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac06f880 con 0x7fbaac155860 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.398+0000 7fbab3f50640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac06f5a0 con 0x7fbaac0c9c70 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.398+0000 7fbab3f50640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fbaac06e8c0 con 0x7fbaac15f650 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 0x7fbaac0c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 0x7fbaac0c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42674/0 (socket says 172.21.15.117:42674) 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 -- 172.21.15.117:0/3227167768 learned_addr learned my addr 172.21.15.117:0/3227167768 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab1cec640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaac0c9c70 0x7fbaac158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab24ed640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbaac15f650 0x7fbaac14f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 -- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbaac15f650 msgr2=0x7fbaac14f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbaac15f650 0x7fbaac14f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 -- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaac0c9c70 msgr2=0x7fbaac158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaac0c9c70 0x7fbaac158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 -- 172.21.15.117:0/3227167768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac1f2830 con 0x7fbaac155860 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab14eb640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 0x7fbaac0c66b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00c390 tx=0x7fba9c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c011790 con 0x7fbaac155860 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba9c017070 con 0x7fbaac155860 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaac1579e0 con 0x7fbaac155860 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c011cc0 con 0x7fbaac155860 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.399+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaac150ce0 con 0x7fbaac155860 2024-09-17T05:27:57.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.400+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaac1f2d30 con 0x7fbaac155860 2024-09-17T05:27:57.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.401+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fba9c01c460 con 0x7fbaac155860 2024-09-17T05:27:57.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.402+0000 7fbaa27fc640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba90078bc0 0x7fba9007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:57.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.402+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fba9c09ac70 con 0x7fbaac155860 2024-09-17T05:27:57.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.402+0000 7fbab1cec640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba90078bc0 0x7fba9007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:57.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.403+0000 7fbab1cec640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba90078bc0 0x7fba9007b0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8004630 tx=0x7fbaa800b2e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:57.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.406+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbaac1f2d30 con 0x7fbaac155860 2024-09-17T05:27:57.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.618+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaac0c9730 con 0x7fbaac155860 2024-09-17T05:27:57.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.618+0000 7fbaa27fc640 1 -- 172.21.15.117:0/3227167768 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7fba9c0627b0 con 0x7fbaac155860 2024-09-17T05:27:57.620 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:27:57.620 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:27:57.620 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:27:57.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba90078bc0 msgr2=0x7fba9007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:57.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba90078bc0 0x7fba9007b0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8004630 tx=0x7fbaa800b2e0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 msgr2=0x7fbaac0c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:57.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 0x7fbaac0c66b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00c390 tx=0x7fba9c00c850 comp rx=0 tx=0).stop 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 shutdown_connections 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba90078bc0 0x7fba9007b0d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbaac15f650 0x7fbaac14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbaac155860 0x7fbaac0c66b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 --2- 172.21.15.117:0/3227167768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaac0c9c70 0x7fbaac158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 >> 172.21.15.117:0/3227167768 conn(0x7fbaac076430 msgr2=0x7fbaac152850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 shutdown_connections 2024-09-17T05:27:57.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:57.620+0000 7fbab3f50640 1 -- 172.21.15.117:0/3227167768 wait complete. 2024-09-17T05:27:58.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T05:27:26.003433+0000', 'modified': '2024-09-17T05:27:55.772725+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4440, 'mds_1': 4434, 'mds_2': 4422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4422': {'gid': 4422, 'name': 'e', 'rank': 2, 'incarnation': 13, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.117:6837/242246295', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 242246295}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 242246295}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4434': {'gid': 4434, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.119:6835/2224210538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2224210538}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2224210538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4440': {'gid': 4440, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6839/2500745218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2500745218}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2500745218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4440, 'qdb_cluster': [4422, 4434, 4440]} 2024-09-17T05:27:58.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-17T05:27:58.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-09-17T05:27:58.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 -- 172.21.15.117:0/1800867118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f405c004800 msgr2=0x7f405c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:58.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1800867118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f405c004800 0x7f405c004c10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4058001690 tx=0x7f405c005830 comp rx=0 tx=0).stop 2024-09-17T05:27:58.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 -- 172.21.15.117:0/1800867118 shutdown_connections 2024-09-17T05:27:58.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1800867118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f405c004800 0x7f405c004c10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 -- 172.21.15.117:0/1800867118 >> 172.21.15.117:0/1800867118 conn(0x7f407006e170 msgr2=0x7f407006e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:58.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 -- 172.21.15.117:0/1800867118 shutdown_connections 2024-09-17T05:27:58.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.352+0000 7f4074fc2640 1 -- 172.21.15.117:0/1800867118 wait complete. 2024-09-17T05:27:58.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.354+0000 7f4074fc2640 1 Processor -- start 2024-09-17T05:27:58.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.354+0000 7f4074fc2640 1 -- start start 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4074fc2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f405c004800 0x7f40701582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4074fc2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40701f2a90 0x7f40700c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4074fc2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 0x7f40700c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4074fc2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f407015f090 con 0x7f40700c8840 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4074fc2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f407015edb0 con 0x7f40701f2a90 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4074fc2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f40700b43e0 con 0x7f405c004800 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f406ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 0x7f40700c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f406ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 0x7f40700c8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42698/0 (socket says 172.21.15.117:42698) 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f406ed9d640 1 -- 172.21.15.117:0/1321072631 learned_addr learned my addr 172.21.15.117:0/1321072631 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f406e59c640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f405c004800 0x7f40701582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.355+0000 7f4065d9b640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40701f2a90 0x7f40700c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f406ed9d640 1 -- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f405c004800 msgr2=0x7f40701582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f406ed9d640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f405c004800 0x7f40701582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f406ed9d640 1 -- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40701f2a90 msgr2=0x7f40700c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f406ed9d640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40701f2a90 0x7f40700c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f406ed9d640 1 -- 172.21.15.117:0/1321072631 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40701f2570 con 0x7f40700c8840 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f406ed9d640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 0x7f40700c8c50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4068005fd0 tx=0x7f4068010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f40677fe640 1 -- 172.21.15.117:0/1321072631 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068003970 con 0x7f40700c8840 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f40677fe640 1 -- 172.21.15.117:0/1321072631 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4068015070 con 0x7f40700c8840 2024-09-17T05:27:58.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40700b5720 con 0x7f40700c8840 2024-09-17T05:27:58.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f40677fe640 1 -- 172.21.15.117:0/1321072631 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068004030 con 0x7f40700c8840 2024-09-17T05:27:58.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.356+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40700c9190 con 0x7f40700c8840 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.358+0000 7f40677fe640 1 -- 172.21.15.117:0/1321072631 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f40680041d0 con 0x7f40700c8840 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.358+0000 7f40677fe640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4050078bc0 0x7f405007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.359+0000 7f40677fe640 1 -- 172.21.15.117:0/1321072631 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f406809ba10 con 0x7f40700c8840 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.359+0000 7f406e59c640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4050078bc0 0x7f405007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.359+0000 7f406e59c640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4050078bc0 0x7f405007b0d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4060001eb0 tx=0x7f4060001e40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.359+0000 7f407c008740 1 Processor -- start 2024-09-17T05:27:58.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.359+0000 7f407c008740 1 -- start start 2024-09-17T05:27:58.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbade4270 0x55edbb1d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbadbb3b0 0x55edbb186940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f407c008740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb18bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f407c008740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55edbb0fe820 con 0x55edbadc3590 2024-09-17T05:27:58.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55edbb0edc50 con 0x55edbade4270 2024-09-17T05:27:58.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55edbb0f1a60 con 0x55edbadbb3b0 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb18bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb18bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42706/0 (socket says 172.21.15.117:42706) 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 -- 172.21.15.117:0/3253946556 learned_addr learned my addr 172.21.15.117:0/3253946556 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406e59c640 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbade4270 0x55edbb1d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f4065d9b640 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbadbb3b0 0x55edbb186940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 -- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbadbb3b0 msgr2=0x55edbb186940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbadbb3b0 0x55edbb186940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 -- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbade4270 msgr2=0x55edbb1d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbade4270 0x55edbb1d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 -- 172.21.15.117:0/3253946556 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbae4f850 con 0x55edbadc3590 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f4065d9b640 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbadbb3b0 0x55edbb186940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f406ed9d640 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb18bb50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4068063570 tx=0x7f406805f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ac5bf52ce45e75 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:58.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f404ffff640 1 -- 172.21.15.117:0/3253946556 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068021070 con 0x55edbadc3590 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f404ffff640 1 -- 172.21.15.117:0/3253946556 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40680635e0 con 0x55edbadc3590 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.361+0000 7f404ffff640 1 -- 172.21.15.117:0/3253946556 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068062a60 con 0x55edbadc3590 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 -- 172.21.15.117:0/3253946556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 msgr2=0x55edbb18bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb18bb50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4068063570 tx=0x7f406805f580 comp rx=0 tx=0).stop 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 -- 172.21.15.117:0/3253946556 shutdown_connections 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb18bb50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbadbb3b0 0x55edbb186940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 --2- 172.21.15.117:0/3253946556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbade4270 0x55edbb1d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 -- 172.21.15.117:0/3253946556 >> 172.21.15.117:0/3253946556 conn(0x55edbadd83c0 msgr2=0x55edbb1cf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 -- 172.21.15.117:0/3253946556 shutdown_connections 2024-09-17T05:27:58.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.362+0000 7f407c008740 1 -- 172.21.15.117:0/3253946556 wait complete. 2024-09-17T05:27:58.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.363+0000 7f407c008740 1 -- start start 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.363+0000 7f407c008740 1 Processor -- start 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1df580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55edbae5e620 con 0x55edbadc3590 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55edbb0935f0 con 0x55edbadbb3b0 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55edbb1bf3f0 con 0x55edbade4270 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 10 client.0 _ll_register_callbacks cb 0x55edbb0970a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1df580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1df580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42708/0 (socket says 172.21.15.117:42708) 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 -- 172.21.15.117:0/586959292 learned_addr learned my addr 172.21.15.117:0/586959292 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f406e59c640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f406ed9d640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 -- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 msgr2=0x55edbb1857d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1857d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 -- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 msgr2=0x55edbb1be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 -- 172.21.15.117:0/586959292 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbaec66b0 con 0x55edbadc3590 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f406ed9d640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f406e59c640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f4065d9b640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1df580 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4058001690 tx=0x7f40580064d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7baefcd04bf58ce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f404d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f404d7fa640 1 -- 172.21.15.117:0/586959292 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058006920 con 0x55edbadc3590 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f404d7fa640 1 -- 172.21.15.117:0/586959292 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4058003a00 con 0x55edbadc3590 2024-09-17T05:27:58.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.365+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40680635e0 con 0x55edbadc3590 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f404d7fa640 1 -- 172.21.15.117:0/586959292 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f4058049050 con 0x55edbadc3590 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 10 client.4863 fetch_fsmap learned FSMap version 14 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x55edbb1c03d0 con 0x55edbadc3590 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f404d7fa640 1 -- 172.21.15.117:0/586959292 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40680635e0 con 0x55edbadc3590 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f404d7fa640 1 -- 172.21.15.117:0/586959292 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) ==== 4760+0+0 (secure 0 0 0) 0x7f405804c020 con 0x55edbadc3590 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 10 client.4863 fetch_fsmap finished waiting for FSMap version 14 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 10 client.4863 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 20 client.4863 populate_metadata read hostname 'smithi117' 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbada2e40 0x55edbb1ea260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 4 client.4863 mds_command: new command op to 4422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 --> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edbb100770 con 0x55edbada2e40 2024-09-17T05:27:58.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.366+0000 7f406e59c640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbada2e40 0x55edbb1ea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:27:58.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.367+0000 7f404d7fa640 10 client.4863 ms_handle_connect on v2:172.21.15.117:6836/242246295 2024-09-17T05:27:58.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:27:58.367+0000 7f406e59c640 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbada2e40 0x55edbb1ea260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e5fc5ed49f81ff6 server_cookie=1137c4cb03d316fe in_seq=0 out_seq=0 2024-09-17T05:28:00.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.778+0000 7f404d7fa640 1 -- 172.21.15.117:0/586959292 <== mds.2 v2:172.21.15.117:6836/242246295 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55edbb100770 con 0x55edbada2e40 2024-09-17T05:28:00.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.778+0000 7f404d7fa640 10 client.4863 handle_command_reply: tid=0 2024-09-17T05:28:00.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 client.4863 shutdown 2024-09-17T05:28:00.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 msgr2=0x55edbb1df580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1df580 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4058001690 tx=0x7f40580064d0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 shutdown_connections 2024-09-17T05:28:00.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbada2e40 0x55edbb1ea260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1df580 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 --2- 172.21.15.117:0/586959292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f404d7fa640 0 client.4863 ms_handle_reset on v2:172.21.15.117:6836/242246295 2024-09-17T05:28:00.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 >> 172.21.15.117:0/586959292 conn(0x55edbadd83c0 msgr2=0x55edbb1c6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 shutdown_connections 2024-09-17T05:28:00.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 1 -- 172.21.15.117:0/586959292 wait complete. 2024-09-17T05:28:00.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.779+0000 7f407c008740 20 client.4863 trim_cache size 0 max 16384 2024-09-17T05:28:00.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.799+0000 7f407c008740 1 Processor -- start 2024-09-17T05:28:00.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.799+0000 7f407c008740 1 -- start start 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.800+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1d8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.800+0000 7f407c008740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1c4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.800+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.800+0000 7f407c008740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f40701fe350 con 0x55edbadc3590 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.800+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f40701fe850 con 0x55edbadbb3b0 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.800+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55edbb1bce30 con 0x55edbade4270 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42718/0 (socket says 172.21.15.117:42718) 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 -- 172.21.15.117:0/411078341 learned_addr learned my addr 172.21.15.117:0/411078341 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f406ed9d640 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f406e59c640 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1d8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 -- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 msgr2=0x55edbb1c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 -- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 msgr2=0x55edbb1d8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1d8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 -- 172.21.15.117:0/411078341 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbb18a500 con 0x55edbadc3590 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f4065d9b640 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1c4440 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4058000fd0 tx=0x7f40580015a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400819bf2785767d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f404f7fe640 1 -- 172.21.15.117:0/411078341 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058006920 con 0x55edbadc3590 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f404f7fe640 1 -- 172.21.15.117:0/411078341 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4058039040 con 0x55edbadc3590 2024-09-17T05:28:00.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.801+0000 7f404f7fe640 1 -- 172.21.15.117:0/411078341 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058003a00 con 0x55edbadc3590 2024-09-17T05:28:00.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 -- 172.21.15.117:0/411078341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 msgr2=0x55edbb1c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1c4440 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4058000fd0 tx=0x7f40580015a0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 -- 172.21.15.117:0/411078341 shutdown_connections 2024-09-17T05:28:00.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb1c4440 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 --2- 172.21.15.117:0/411078341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb1d8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.802+0000 7f407c008740 1 -- 172.21.15.117:0/411078341 >> 172.21.15.117:0/411078341 conn(0x55edbadd83c0 msgr2=0x55edbb18d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.803+0000 7f407c008740 1 -- 172.21.15.117:0/411078341 shutdown_connections 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.803+0000 7f407c008740 1 -- 172.21.15.117:0/411078341 wait complete. 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.803+0000 7f407c008740 1 -- start start 2024-09-17T05:28:00.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.803+0000 7f407c008740 1 Processor -- start 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb24f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb245f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55edbb1bca50 con 0x55edbadc3590 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55edbb185d60 con 0x55edbadbb3b0 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55edbb1db8e0 con 0x55edbade4270 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f407c008740 10 client.0 _ll_register_callbacks cb 0x7f40680093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f4065d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb245f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f4065d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb245f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42720/0 (socket says 172.21.15.117:42720) 2024-09-17T05:28:00.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f4065d9b640 1 -- 172.21.15.117:0/1642981338 learned_addr learned my addr 172.21.15.117:0/1642981338 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f406e59c640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb24f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.805+0000 7f406ed9d640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f4065d9b640 1 -- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 msgr2=0x55edbb1c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f4065d9b640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f4065d9b640 1 -- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 msgr2=0x55edbb24f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f4065d9b640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb24f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f4065d9b640 1 -- 172.21.15.117:0/1642981338 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbb1c6760 con 0x55edbadc3590 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f4065d9b640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb245f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f405802a8d0 tx=0x7f4058006b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0e5b46f4dd1c8c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f404d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f404d7fa640 1 -- 172.21.15.117:0/1642981338 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058044430 con 0x55edbadc3590 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f404d7fa640 1 -- 172.21.15.117:0/1642981338 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f40580445d0 con 0x55edbadc3590 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4058039040 con 0x55edbadc3590 2024-09-17T05:28:00.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f404d7fa640 1 -- 172.21.15.117:0/1642981338 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f405805e050 con 0x55edbadc3590 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f407c008740 10 client.4869 fetch_fsmap learned FSMap version 14 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x55edbb24cf80 con 0x55edbadc3590 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.806+0000 7f404d7fa640 1 -- 172.21.15.117:0/1642981338 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4058039040 con 0x55edbadc3590 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f404d7fa640 1 -- 172.21.15.117:0/1642981338 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) ==== 4760+0+0 (secure 0 0 0) 0x7f4058061020 con 0x55edbadc3590 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f407c008740 10 client.4869 fetch_fsmap finished waiting for FSMap version 14 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f407c008740 10 client.4869 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f407c008740 20 client.4869 populate_metadata read hostname 'smithi117' 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f407c008740 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbb1caed0 0x55edbb258a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f407c008740 4 client.4869 mds_command: new command op to 4422 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 --> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] -- command(tid 0: {"prefix": "status"}) -- 0x55edbb1bdfe0 con 0x55edbb1caed0 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f406e59c640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbb1caed0 0x55edbb258a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f404d7fa640 10 client.4869 ms_handle_connect on v2:172.21.15.117:6836/242246295 2024-09-17T05:28:00.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.807+0000 7f406e59c640 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbb1caed0 0x55edbb258a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=71f38d4b91b63774 server_cookie=5af99311026799d4 in_seq=0 out_seq=0 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f404d7fa640 1 -- 172.21.15.117:0/1642981338 <== mds.2 v2:172.21.15.117:6836/242246295 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55edbb1bdfe0 con 0x55edbb1caed0 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f404d7fa640 10 client.4869 handle_command_reply: tid=0 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 client.4869 shutdown 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 msgr2=0x55edbb245f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb245f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f405802a8d0 tx=0x7f4058006b90 comp rx=0 tx=0).stop 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 shutdown_connections 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:6836/242246295,v1:172.21.15.117:6837/242246295] conn(0x55edbb1caed0 0x55edbb258a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55edbade4270 0x55edbb1c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55edbadc3590 0x55edbb245f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 --2- 172.21.15.117:0/1642981338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55edbadbb3b0 0x55edbb24f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f404d7fa640 0 client.4869 ms_handle_reset on v2:172.21.15.117:6836/242246295 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 >> 172.21.15.117:0/1642981338 conn(0x55edbadd83c0 msgr2=0x55edbb1c0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 shutdown_connections 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 1 -- 172.21.15.117:0/1642981338 wait complete. 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.808+0000 7f407c008740 20 client.4869 trim_cache size 0 max 16384 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "8527add3-c46f-4f7f-86e8-5cd706a0043e", 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 2, 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 4422, 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-09-17T05:28:00.814 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-09-17T05:28:00.815 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 11.048632837, 2024-09-17T05:28:00.815 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 14, 2024-09-17T05:28:00.815 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 25, 2024-09-17T05:28:00.815 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 25, 2024-09-17T05:28:00.815 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 40.196284161999998 2024-09-17T05:28:00.815 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4050078bc0 msgr2=0x7f405007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4050078bc0 0x7f405007b0d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4060001eb0 tx=0x7f4060001e40 comp rx=0 tx=0).stop 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 msgr2=0x7f40700c8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 0x7f40700c8c50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4068005fd0 tx=0x7f4068010f40 comp rx=0 tx=0).stop 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 shutdown_connections 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4050078bc0 0x7f405007b0d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40700c8840 0x7f40700c8c50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40701f2a90 0x7f40700c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 --2- 172.21.15.117:0/1321072631 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f405c004800 0x7f40701582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 >> 172.21.15.117:0/1321072631 conn(0x7f407006e170 msgr2=0x7f407015af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 shutdown_connections 2024-09-17T05:28:00.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.809+0000 7f4074fc2640 1 -- 172.21.15.117:0/1321072631 wait complete. 2024-09-17T05:28:00.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-17T05:28:00.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 -- 172.21.15.117:0/3598352326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c90a0 msgr2=0x7fecc80c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 --2- 172.21.15.117:0/3598352326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c90a0 0x7fecc80c9960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fecc80b3ba0 tx=0x7fecbc02a560 comp rx=0 tx=0).stop 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 -- 172.21.15.117:0/3598352326 shutdown_connections 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 --2- 172.21.15.117:0/3598352326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c90a0 0x7fecc80c9960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 --2- 172.21.15.117:0/3598352326 >> v2:172.21.15.119:3300/0 conn(0x7fecc80c94b0 0x7fecc80c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 --2- 172.21.15.117:0/3598352326 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fecc80736a0 0x7fecc81530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 -- 172.21.15.117:0/3598352326 >> 172.21.15.117:0/3598352326 conn(0x7fecc806d3d0 msgr2=0x7fecc8072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 -- 172.21.15.117:0/3598352326 shutdown_connections 2024-09-17T05:28:00.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.941+0000 7fecce3c2640 1 -- 172.21.15.117:0/3598352326 wait complete. 2024-09-17T05:28:00.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.942+0000 7fecce3c2640 1 Processor -- start 2024-09-17T05:28:00.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.943+0000 7fecce3c2640 1 -- start start 2024-09-17T05:28:00.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecce3c2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fecc80736a0 0x7fecc81f0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecce3c2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc80c90a0 0x7fecc814e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecce3c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 0x7fecc8145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecce3c2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8159a70 con 0x7fecc80c94b0 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecce3c2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8159790 con 0x7fecc80c90a0 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecce3c2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fecc8159d50 con 0x7fecc80736a0 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 0x7fecc8145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 0x7fecc8145fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42734/0 (socket says 172.21.15.117:42734) 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 -- 172.21.15.117:0/2844121782 learned_addr learned my addr 172.21.15.117:0/2844121782 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecc77fe640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc80c90a0 0x7fecc814e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7fecc7fff640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fecc80736a0 0x7fecc81f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 -- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fecc80736a0 msgr2=0x7fecc81f0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fecc80736a0 0x7fecc81f0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 -- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc80c90a0 msgr2=0x7fecc814e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc80c90a0 0x7fecc814e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 -- 172.21.15.117:0/2844121782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc815c210 con 0x7fecc80c94b0 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.944+0000 7feccc95f640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 0x7fecc8145fa0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fecc000e9c0 tx=0x7fecc000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.945+0000 7fecc4ff9640 1 -- 172.21.15.117:0/2844121782 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0004d20 con 0x7fecc80c94b0 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.945+0000 7fecc4ff9640 1 -- 172.21.15.117:0/2844121782 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecc000cb20 con 0x7fecc80c94b0 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.945+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc81e3be0 con 0x7fecc80c94b0 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.945+0000 7fecc4ff9640 1 -- 172.21.15.117:0/2844121782 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0005020 con 0x7fecc80c94b0 2024-09-17T05:28:00.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.945+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc81e41a0 con 0x7fecc80c94b0 2024-09-17T05:28:00.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.946+0000 7fecc4ff9640 1 -- 172.21.15.117:0/2844121782 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fecc00051c0 con 0x7fecc80c94b0 2024-09-17T05:28:00.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.947+0000 7fecc4ff9640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fecb4078bc0 0x7fecb407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.947+0000 7fecc4ff9640 1 -- 172.21.15.117:0/2844121782 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fecc009b920 con 0x7fecc80c94b0 2024-09-17T05:28:00.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.947+0000 7fecc7fff640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fecb4078bc0 0x7fecb407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.948+0000 7fecc7fff640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fecb4078bc0 0x7fecb407b0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fecb8005fd0 tx=0x7fecb800b260 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.948+0000 7fecd5408740 1 Processor -- start 2024-09-17T05:28:00.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.948+0000 7fecd5408740 1 -- start start 2024-09-17T05:28:00.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecd5408740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da351e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9d9f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9d9f1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecd5408740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eb9d963820 con 0x55eb9d649270 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55eb9d952c50 con 0x55eb9d628590 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55eb9d956a60 con 0x55eb9d6203b0 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da351e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da351e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42750/0 (socket says 172.21.15.117:42750) 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 -- 172.21.15.117:0/2529573201 learned_addr learned my addr 172.21.15.117:0/2529573201 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc77fe640 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9d9f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7feccc95f640 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9d9f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 -- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 msgr2=0x55eb9d9f0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9d9f0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 -- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 msgr2=0x55eb9d9f1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9d9f1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.949+0000 7fecc7fff640 1 -- 172.21.15.117:0/2529573201 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb9d6b4850 con 0x55eb9d649270 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.950+0000 7fecc7fff640 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da351e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fecb800bb20 tx=0x7fecb8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3671a3ade6e6bf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.950+0000 7fecadffb640 1 -- 172.21.15.117:0/2529573201 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8011d40 con 0x55eb9d649270 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.950+0000 7fecadffb640 1 -- 172.21.15.117:0/2529573201 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecb801e420 con 0x55eb9d649270 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.950+0000 7fecadffb640 1 -- 172.21.15.117:0/2529573201 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb801e7f0 con 0x55eb9d649270 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.950+0000 7fecd5408740 1 -- 172.21.15.117:0/2529573201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 msgr2=0x55eb9da351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.950+0000 7fecd5408740 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da351e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fecb800bb20 tx=0x7fecb8013000 comp rx=0 tx=0).stop 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 -- 172.21.15.117:0/2529573201 shutdown_connections 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9d9f1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9d9f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 --2- 172.21.15.117:0/2529573201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da351e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 -- 172.21.15.117:0/2529573201 >> 172.21.15.117:0/2529573201 conn(0x55eb9d63d3c0 msgr2=0x55eb9da34920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 -- 172.21.15.117:0/2529573201 shutdown_connections 2024-09-17T05:28:00.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 -- 172.21.15.117:0/2529573201 wait complete. 2024-09-17T05:28:00.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.951+0000 7fecd5408740 1 -- start start 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.952+0000 7fecd5408740 1 Processor -- start 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d6203b0 0x55eb9da48910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9d9ea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eb9d6c3620 con 0x55eb9d649270 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55eb9d8f85f0 con 0x55eb9d6203b0 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55eb9da2ddb0 con 0x55eb9d628590 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecd5408740 10 client.0 _ll_register_callbacks cb 0x55eb9d8fc0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9d9ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9d9ea340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42760/0 (socket says 172.21.15.117:42760) 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 -- 172.21.15.117:0/4249981900 learned_addr learned my addr 172.21.15.117:0/4249981900 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecc77fe640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7fecc7fff640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d6203b0 0x55eb9da48910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 -- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d628590 msgr2=0x55eb9da1e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 -- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d6203b0 msgr2=0x55eb9da48910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d6203b0 0x55eb9da48910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 -- 172.21.15.117:0/4249981900 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb9d72b6b0 con 0x55eb9d649270 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feccc95f640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9d9ea340 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fecc000e990 tx=0x7fecc00638a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1541c9004edfb31a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feca6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feca6ffd640 1 -- 172.21.15.117:0/4249981900 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0089070 con 0x55eb9d649270 2024-09-17T05:28:00.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.953+0000 7feca6ffd640 1 -- 172.21.15.117:0/4249981900 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fecc005ed10 con 0x55eb9d649270 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb801e420 con 0x55eb9d649270 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7feca6ffd640 1 -- 172.21.15.117:0/4249981900 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7fecc001f040 con 0x55eb9d649270 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 10 client.4881 fetch_fsmap learned FSMap version 14 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x55eb9d9eb250 con 0x55eb9d649270 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7feca6ffd640 1 -- 172.21.15.117:0/4249981900 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecb801e420 con 0x55eb9d649270 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7feca6ffd640 1 -- 172.21.15.117:0/4249981900 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) ==== 4760+0+0 (secure 0 0 0) 0x7fecc000f040 con 0x55eb9d649270 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 10 client.4881 fetch_fsmap finished waiting for FSMap version 14 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 10 client.4881 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 20 client.4881 populate_metadata read hostname 'smithi117' 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9d607e40 0x55eb9da539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 4 client.4881 mds_command: new command op to 4434 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 --> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb9d965770 con 0x55eb9d607e40 2024-09-17T05:28:00.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.954+0000 7fecc7fff640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9d607e40 0x55eb9da539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.955+0000 7fecc7fff640 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9d607e40 0x55eb9da539d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74342ecc4b09ec0e server_cookie=78d9b0b9d45e07f9 in_seq=0 out_seq=0 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.955+0000 7feca6ffd640 10 client.4881 ms_handle_connect on v2:172.21.15.119:6834/2224210538 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.955+0000 7feca6ffd640 1 -- 172.21.15.117:0/4249981900 <== mds.1 v2:172.21.15.119:6834/2224210538 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55eb9d965770 con 0x55eb9d607e40 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.955+0000 7feca6ffd640 10 client.4881 handle_command_reply: tid=0 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.955+0000 7fecd5408740 1 client.4881 shutdown 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 msgr2=0x55eb9d9ea340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9d9ea340 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fecc000e990 tx=0x7fecc00638a0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 shutdown_connections 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9d607e40 0x55eb9da539d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9d9ea340 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 --2- 172.21.15.117:0/4249981900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d6203b0 0x55eb9da48910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7feca6ffd640 0 client.4881 ms_handle_reset on v2:172.21.15.119:6834/2224210538 2024-09-17T05:28:00.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 >> 172.21.15.117:0/4249981900 conn(0x55eb9d63d3c0 msgr2=0x55eb9da27940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 shutdown_connections 2024-09-17T05:28:00.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 1 -- 172.21.15.117:0/4249981900 wait complete. 2024-09-17T05:28:00.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.956+0000 7fecd5408740 20 client.4881 trim_cache size 0 max 16384 2024-09-17T05:28:00.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.974+0000 7fecd5408740 1 Processor -- start 2024-09-17T05:28:00.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.974+0000 7fecd5408740 1 -- start start 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da44e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecd5408740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 0x55eb9da453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d649270 0x55eb9da1dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecd5408740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81f40e0 con 0x55eb9d628590 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55eb9da49060 con 0x55eb9d649270 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55eb9da2dae0 con 0x55eb9d6203b0 2024-09-17T05:28:00.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecc77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 0x55eb9da453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecc77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 0x55eb9da453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42774/0 (socket says 172.21.15.117:42774) 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecc77fe640 1 -- 172.21.15.117:0/3546136957 learned_addr learned my addr 172.21.15.117:0/3546136957 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7fecc7fff640 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da44e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.975+0000 7feccc95f640 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d649270 0x55eb9da1dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecc77fe640 1 -- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 msgr2=0x55eb9da44e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecc77fe640 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da44e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecc77fe640 1 -- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d649270 msgr2=0x55eb9da1dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecc77fe640 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d649270 0x55eb9da1dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecc77fe640 1 -- 172.21.15.117:0/3546136957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb9da35490 con 0x55eb9d628590 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecc77fe640 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 0x55eb9da453d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fecbc000c00 tx=0x7fecbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36698f67bee9fe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecad7fa640 1 -- 172.21.15.117:0/3546136957 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc035040 con 0x55eb9d628590 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecad7fa640 1 -- 172.21.15.117:0/3546136957 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecbc004320 con 0x55eb9d628590 2024-09-17T05:28:00.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.976+0000 7fecad7fa640 1 -- 172.21.15.117:0/3546136957 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc003850 con 0x55eb9d628590 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 -- 172.21.15.117:0/3546136957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 msgr2=0x55eb9da453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 0x55eb9da453d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fecbc000c00 tx=0x7fecbc0046c0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 -- 172.21.15.117:0/3546136957 shutdown_connections 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d649270 0x55eb9da1dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d628590 0x55eb9da453d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 --2- 172.21.15.117:0/3546136957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da44e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 -- 172.21.15.117:0/3546136957 >> 172.21.15.117:0/3546136957 conn(0x55eb9d63d3c0 msgr2=0x55eb9da34d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 -- 172.21.15.117:0/3546136957 shutdown_connections 2024-09-17T05:28:00.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 -- 172.21.15.117:0/3546136957 wait complete. 2024-09-17T05:28:00.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.977+0000 7fecd5408740 1 -- start start 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.978+0000 7fecd5408740 1 Processor -- start 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da2ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da3dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eb9da250e0 con 0x55eb9d649270 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55eb9da24a90 con 0x55eb9d628590 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x55eb9da2ac00 con 0x55eb9d6203b0 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecd5408740 10 client.0 _ll_register_callbacks cb 0x7fecc005f330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da3dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da3dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42790/0 (socket says 172.21.15.117:42790) 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 -- 172.21.15.117:0/3977170939 learned_addr learned my addr 172.21.15.117:0/3977170939 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecc77fe640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7fecc7fff640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da2ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 -- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 msgr2=0x55eb9da2ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da2ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 -- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 msgr2=0x55eb9da1e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 -- 172.21.15.117:0/3977170939 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb9da1ae20 con 0x55eb9d649270 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feccc95f640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da3dbf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fecc00630c0 tx=0x7fecc000e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9482600824d63e30 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.979+0000 7feca6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7feca6ffd640 1 -- 172.21.15.117:0/3977170939 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0089040 con 0x55eb9d649270 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7feca6ffd640 1 -- 172.21.15.117:0/3977170939 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fecc0067aa0 con 0x55eb9d649270 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecbc004320 con 0x55eb9d649270 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7feca6ffd640 1 -- 172.21.15.117:0/3977170939 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7fecc0062b30 con 0x55eb9d649270 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 10 client.4887 fetch_fsmap learned FSMap version 14 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x55eb9da24c10 con 0x55eb9d649270 2024-09-17T05:28:00.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7feca6ffd640 1 -- 172.21.15.117:0/3977170939 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecbc004320 con 0x55eb9d649270 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7feca6ffd640 1 -- 172.21.15.117:0/3977170939 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) ==== 4760+0+0 (secure 0 0 0) 0x7fecc0062290 con 0x55eb9d649270 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 10 client.4887 fetch_fsmap finished waiting for FSMap version 14 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 10 client.4887 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 20 client.4887 populate_metadata read hostname 'smithi117' 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9daafa00 0x55eb9da29770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 4 client.4887 mds_command: new command op to 4434 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 --> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] -- command(tid 0: {"prefix": "status"}) -- 0x55eb9d961220 con 0x55eb9daafa00 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.980+0000 7fecc7fff640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9daafa00 0x55eb9da29770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7feca6ffd640 10 client.4887 ms_handle_connect on v2:172.21.15.119:6834/2224210538 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecc7fff640 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9daafa00 0x55eb9da29770 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc21304cfd52bf22 server_cookie=d555a4bbfe113da9 in_seq=0 out_seq=0 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7feca6ffd640 1 -- 172.21.15.117:0/3977170939 <== mds.1 v2:172.21.15.119:6834/2224210538 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55eb9d961220 con 0x55eb9daafa00 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7feca6ffd640 10 client.4887 handle_command_reply: tid=0 2024-09-17T05:28:00.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 client.4887 shutdown 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 msgr2=0x55eb9da3dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da3dbf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fecc00630c0 tx=0x7fecc000e8e0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 shutdown_connections 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:6834/2224210538,v1:172.21.15.119:6835/2224210538] conn(0x55eb9daafa00 0x55eb9da29770 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb9d649270 0x55eb9da3dbf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb9d628590 0x55eb9da1e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7feca6ffd640 0 client.4887 ms_handle_reset on v2:172.21.15.119:6834/2224210538 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 --2- 172.21.15.117:0/3977170939 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x55eb9d6203b0 0x55eb9da2ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.981+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 >> 172.21.15.117:0/3977170939 conn(0x55eb9d63d3c0 msgr2=0x55eb9da34d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 shutdown_connections 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecd5408740 1 -- 172.21.15.117:0/3977170939 wait complete. 2024-09-17T05:28:00.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecd5408740 20 client.4887 trim_cache size 0 max 16384 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "8527add3-c46f-4f7f-86e8-5cd706a0043e", 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 1, 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 4434, 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 17.248529434999998, 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 14, 2024-09-17T05:28:00.989 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 25, 2024-09-17T05:28:00.990 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 25, 2024-09-17T05:28:00.990 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 40.350519806000001 2024-09-17T05:28:00.990 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:28:00.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fecb4078bc0 msgr2=0x7fecb407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fecb4078bc0 0x7fecb407b0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fecb8005fd0 tx=0x7fecb800b260 comp rx=0 tx=0).stop 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 msgr2=0x7fecc8145fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 0x7fecc8145fa0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fecc000e9c0 tx=0x7fecc000ee80 comp rx=0 tx=0).stop 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 shutdown_connections 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fecb4078bc0 0x7fecb407b0d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecc80c94b0 0x7fecc8145fa0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc80c90a0 0x7fecc814e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 --2- 172.21.15.117:0/2844121782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fecc80736a0 0x7fecc81f0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 >> 172.21.15.117:0/2844121782 conn(0x7fecc806d3d0 msgr2=0x7fecc814b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:00.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 shutdown_connections 2024-09-17T05:28:00.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:00.982+0000 7fecce3c2640 1 -- 172.21.15.117:0/2844121782 wait complete. 2024-09-17T05:28:00.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-09-17T05:28:01.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1395653756 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 msgr2=0x7f2a3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1395653756 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a3000b9c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2a48072830 tx=0x7f2a30005830 comp rx=0 tx=0).stop 2024-09-17T05:28:01.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1395653756 shutdown_connections 2024-09-17T05:28:01.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1395653756 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a3000b9c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1395653756 >> 172.21.15.117:0/1395653756 conn(0x7f2a4806d3d0 msgr2=0x7f2a48072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1395653756 shutdown_connections 2024-09-17T05:28:01.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.108+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1395653756 wait complete. 2024-09-17T05:28:01.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.109+0000 7f2a4e63f640 1 Processor -- start 2024-09-17T05:28:01.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.110+0000 7f2a4e63f640 1 -- start start 2024-09-17T05:28:01.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4e63f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a4815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4e63f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a481563a0 0x7f2a48153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4e63f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a481567b0 0x7f2a481541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4e63f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a48159a70 con 0x7f2a30005250 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4e63f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4815a0c0 con 0x7f2a481567b0 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4e63f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2a48159d50 con 0x7f2a481563a0 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a4815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a4815a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42808/0 (socket says 172.21.15.117:42808) 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 -- 172.21.15.117:0/1472853879 learned_addr learned my addr 172.21.15.117:0/1472853879 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a481563a0 0x7f2a48153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a481567b0 0x7f2a481541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 -- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a481563a0 msgr2=0x7f2a48153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a481563a0 0x7f2a48153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 -- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a481567b0 msgr2=0x7f2a481541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a481567b0 0x7f2a481541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 -- 172.21.15.117:0/1472853879 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4815ad00 con 0x7f2a30005250 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.111+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a4815a7c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2a38003f30 tx=0x7f2a38001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.112+0000 7f2a44ff9640 1 -- 172.21.15.117:0/1472853879 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38016030 con 0x7f2a30005250 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.112+0000 7f2a44ff9640 1 -- 172.21.15.117:0/1472853879 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a38011640 con 0x7f2a30005250 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.112+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a480c6d10 con 0x7f2a30005250 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.112+0000 7f2a44ff9640 1 -- 172.21.15.117:0/1472853879 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38011a90 con 0x7f2a30005250 2024-09-17T05:28:01.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.112+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a480c7290 con 0x7f2a30005250 2024-09-17T05:28:01.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.113+0000 7f2a44ff9640 1 -- 172.21.15.117:0/1472853879 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a38019070 con 0x7f2a30005250 2024-09-17T05:28:01.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.114+0000 7f2a44ff9640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a34078bc0 0x7f2a3407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.114+0000 7f2a44ff9640 1 -- 172.21.15.117:0/1472853879 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a3809b500 con 0x7f2a30005250 2024-09-17T05:28:01.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.114+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a34078bc0 0x7f2a3407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.115+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a34078bc0 0x7f2a3407b0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2a48150df0 tx=0x7f2a3c005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.115+0000 7f2a5568d740 1 Processor -- start 2024-09-17T05:28:01.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.115+0000 7f2a5568d740 1 -- start start 2024-09-17T05:28:01.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 0x561d3b6ea1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2d53b0 0x561d3b6a0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2dd590 0x561d3b6a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a5568d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561d3b618820 con 0x561d3b2fe270 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561d3b607c50 con 0x561d3b2d53b0 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x561d3b60ba60 con 0x561d3b2dd590 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 0x561d3b6ea1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 0x561d3b6ea1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42816/0 (socket says 172.21.15.117:42816) 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a47fff640 1 -- 172.21.15.117:0/2407867451 learned_addr learned my addr 172.21.15.117:0/2407867451 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2dd590 0x561d3b6a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.116+0000 7f2a477fe640 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2d53b0 0x561d3b6a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a47fff640 1 -- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2dd590 msgr2=0x561d3b6a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a47fff640 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2dd590 0x561d3b6a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a47fff640 1 -- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2d53b0 msgr2=0x561d3b6a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a47fff640 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2d53b0 0x561d3b6a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a47fff640 1 -- 172.21.15.117:0/2407867451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3b369850 con 0x561d3b2fe270 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a47fff640 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 0x561d3b6ea1e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2a38062fb0 tx=0x7f2a38062850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba960914b62da2f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a2dffb640 1 -- 172.21.15.117:0/2407867451 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38089070 con 0x561d3b2fe270 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a2dffb640 1 -- 172.21.15.117:0/2407867451 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a38063720 con 0x561d3b2fe270 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.117+0000 7f2a2dffb640 1 -- 172.21.15.117:0/2407867451 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38063020 con 0x561d3b2fe270 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 -- 172.21.15.117:0/2407867451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 msgr2=0x561d3b6ea1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 0x561d3b6ea1e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2a38062fb0 tx=0x7f2a38062850 comp rx=0 tx=0).stop 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 -- 172.21.15.117:0/2407867451 shutdown_connections 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2dd590 0x561d3b6a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2d53b0 0x561d3b6a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 --2- 172.21.15.117:0/2407867451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2fe270 0x561d3b6ea1e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 -- 172.21.15.117:0/2407867451 >> 172.21.15.117:0/2407867451 conn(0x561d3b2f23c0 msgr2=0x561d3b6e9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 -- 172.21.15.117:0/2407867451 shutdown_connections 2024-09-17T05:28:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.118+0000 7f2a5568d740 1 -- 172.21.15.117:0/2407867451 wait complete. 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.119+0000 7f2a5568d740 1 -- start start 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.119+0000 7f2a5568d740 1 Processor -- start 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b6d67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b6de680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2fe270 0x561d3b6dbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561d3b378620 con 0x561d3b2d53b0 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561d3b5ad5f0 con 0x561d3b2dd590 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x561d3b6f5b00 con 0x561d3b2fe270 2024-09-17T05:28:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a5568d740 10 client.0 _ll_register_callbacks cb 0x561d3b5b10a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b6d67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b6d67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42822/0 (socket says 172.21.15.117:42822) 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 -- 172.21.15.117:0/1153872757 learned_addr learned my addr 172.21.15.117:0/1153872757 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b6de680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2fe270 0x561d3b6dbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 -- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2fe270 msgr2=0x561d3b6dbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2fe270 0x561d3b6dbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 -- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 msgr2=0x561d3b6de680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b6de680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a47fff640 1 -- 172.21.15.117:0/1153872757 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d3b3e06b0 con 0x561d3b2d53b0 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.120+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2fe270 0x561d3b6dbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b6d67d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2a38003f00 tx=0x7f2a3805f760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda694dbb056bd62 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1153872757 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38089050 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1153872757 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a3806c720 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a38063720 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1153872757 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f2a38063020 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 10 client.4899 fetch_fsmap learned FSMap version 14 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x561d3b6f5810 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1153872757 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a38063720 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1153872757 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) ==== 4760+0+0 (secure 0 0 0) 0x7f2a38023060 con 0x561d3b2d53b0 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 10 client.4899 fetch_fsmap finished waiting for FSMap version 14 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 10 client.4899 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-17T05:28:01.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 20 client.4899 populate_metadata read hostname 'smithi117' 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b2bce40 0x561d3b7044a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 4 client.4899 mds_command: new command op to 4440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 --> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d3b61a770 con 0x561d3b2bce40 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.121+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b2bce40 0x561d3b7044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.122+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b2bce40 0x561d3b7044a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80c5079d8bdc1ba8 server_cookie=fedd929d0e16d198 in_seq=0 out_seq=0 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.122+0000 7f2a26ffd640 10 client.4899 ms_handle_connect on v2:172.21.15.119:6838/2500745218 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1153872757 <== mds.0 v2:172.21.15.119:6838/2500745218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561d3b61a770 con 0x561d3b2bce40 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a26ffd640 10 client.4899 handle_command_reply: tid=0 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 client.4899 shutdown 2024-09-17T05:28:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 msgr2=0x561d3b6d67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b6d67d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2a38003f00 tx=0x7f2a3805f760 comp rx=0 tx=0).stop 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 shutdown_connections 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b2bce40 0x561d3b7044a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2fe270 0x561d3b6dbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a26ffd640 0 client.4899 ms_handle_reset on v2:172.21.15.119:6838/2500745218 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b6de680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1153872757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b6d67d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 >> 172.21.15.117:0/1153872757 conn(0x561d3b2f23c0 msgr2=0x561d3b6e0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 shutdown_connections 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 1 -- 172.21.15.117:0/1153872757 wait complete. 2024-09-17T05:28:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.123+0000 7f2a5568d740 20 client.4899 trim_cache size 0 max 16384 2024-09-17T05:28:01.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.141+0000 7f2a5568d740 1 Processor -- start 2024-09-17T05:28:01.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.141+0000 7f2a5568d740 1 -- start start 2024-09-17T05:28:01.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2bce40 0x561d3b6a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2d53b0 0x561d3b6a5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 0x561d3b6d81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a5568d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a481fa1c0 con 0x561d3b2dd590 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561d3b5ad5f0 con 0x561d3b2bce40 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x561d3b378620 con 0x561d3b2d53b0 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a4cbdc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 0x561d3b6d81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a4cbdc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 0x561d3b6d81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42828/0 (socket says 172.21.15.117:42828) 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a4cbdc640 1 -- 172.21.15.117:0/3847054507 learned_addr learned my addr 172.21.15.117:0/3847054507 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.142+0000 7f2a477fe640 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2d53b0 0x561d3b6a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a47fff640 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2bce40 0x561d3b6a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a4cbdc640 1 -- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2d53b0 msgr2=0x561d3b6a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2d53b0 0x561d3b6a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a4cbdc640 1 -- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2bce40 msgr2=0x561d3b6a4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2bce40 0x561d3b6a4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a4cbdc640 1 -- 172.21.15.117:0/3847054507 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3b6a68c0 con 0x561d3b2dd590 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a47fff640 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2bce40 0x561d3b6a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 0x561d3b6d81d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2a40001490 tx=0x7f2a4815a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea49611b4621c7b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a2d7fa640 1 -- 172.21.15.117:0/3847054507 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4004d070 con 0x561d3b2dd590 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a2d7fa640 1 -- 172.21.15.117:0/3847054507 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a40003990 con 0x561d3b2dd590 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.143+0000 7f2a2d7fa640 1 -- 172.21.15.117:0/3847054507 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a40040040 con 0x561d3b2dd590 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 -- 172.21.15.117:0/3847054507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 msgr2=0x561d3b6d81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 0x561d3b6d81d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2a40001490 tx=0x7f2a4815a030 comp rx=0 tx=0).stop 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 -- 172.21.15.117:0/3847054507 shutdown_connections 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2dd590 0x561d3b6d81d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2d53b0 0x561d3b6a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 --2- 172.21.15.117:0/3847054507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2bce40 0x561d3b6a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 -- 172.21.15.117:0/3847054507 >> 172.21.15.117:0/3847054507 conn(0x561d3b2f23c0 msgr2=0x561d3b6a7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 -- 172.21.15.117:0/3847054507 shutdown_connections 2024-09-17T05:28:01.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 -- 172.21.15.117:0/3847054507 wait complete. 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.144+0000 7f2a5568d740 1 -- start start 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.145+0000 7f2a5568d740 1 Processor -- start 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2bce40 0x561d3b766350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b766890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b75ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561d3b6df770 con 0x561d3b2d53b0 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561d3b6df120 con 0x561d3b2dd590 2024-09-17T05:28:01.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x561d3b6f8f20 con 0x561d3b2bce40 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a5568d740 10 client.0 _ll_register_callbacks cb 0x7f2a3804d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b766890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b766890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42830/0 (socket says 172.21.15.117:42830) 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 -- 172.21.15.117:0/1471273688 learned_addr learned my addr 172.21.15.117:0/1471273688 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2bce40 0x561d3b766350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a4cbdc640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b75ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 -- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2bce40 msgr2=0x561d3b766350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2bce40 0x561d3b766350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 -- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 msgr2=0x561d3b75ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b75ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.146+0000 7f2a477fe640 1 -- 172.21.15.117:0/1471273688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d3b6ded60 con 0x561d3b2d53b0 2024-09-17T05:28:01.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a477fe640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b766890 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c012f40 tx=0x7f2a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbc47e35d36905d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1471273688 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c037030 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1471273688 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2a3c018030 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a40003990 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1471273688 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f2a3c040050 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 10 client.4905 fetch_fsmap learned FSMap version 14 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x561d3b766e50 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1471273688 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a40003990 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1471273688 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) ==== 4760+0+0 (secure 0 0 0) 0x7f2a3c043020 con 0x561d3b2d53b0 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 10 client.4905 fetch_fsmap finished waiting for FSMap version 14 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 10 client.4905 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 20 client.4905 populate_metadata read hostname 'smithi117' 2024-09-17T05:28:01.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b292280 0x561d3b772a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 4 client.4905 mds_command: new command op to 4440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.147+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 --> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] -- command(tid 0: {"prefix": "status"}) -- 0x561d3b616220 con 0x561d3b292280 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b292280 0x561d3b772a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a26ffd640 10 client.4905 ms_handle_connect on v2:172.21.15.119:6838/2500745218 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a47fff640 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b292280 0x561d3b772a90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d35f041cb8b01c2 server_cookie=1b6fb71718fc5117 in_seq=0 out_seq=0 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a26ffd640 1 -- 172.21.15.117:0/1471273688 <== mds.0 v2:172.21.15.119:6838/2500745218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x561d3b616220 con 0x561d3b292280 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a26ffd640 10 client.4905 handle_command_reply: tid=0 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 client.4905 shutdown 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 msgr2=0x561d3b766890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b766890 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c012f40 tx=0x7f2a3c0046c0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 shutdown_connections 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:6838/2500745218,v1:172.21.15.119:6839/2500745218] conn(0x561d3b292280 0x561d3b772a90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d3b2dd590 0x561d3b75ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561d3b2d53b0 0x561d3b766890 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a26ffd640 0 client.4905 ms_handle_reset on v2:172.21.15.119:6838/2500745218 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 --2- 172.21.15.117:0/1471273688 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x561d3b2bce40 0x561d3b766350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 >> 172.21.15.117:0/1471273688 conn(0x561d3b2f23c0 msgr2=0x561d3b6a7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 shutdown_connections 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.148+0000 7f2a5568d740 1 -- 172.21.15.117:0/1471273688 wait complete. 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a5568d740 20 client.4905 trim_cache size 0 max 16384 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "8527add3-c46f-4f7f-86e8-5cd706a0043e", 2024-09-17T05:28:01.156 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 0, 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 4440, 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 34.513093830999999, 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 14, 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 25, 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 25, 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 40.513674795 2024-09-17T05:28:01.157 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:28:01.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a34078bc0 msgr2=0x7f2a3407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a34078bc0 0x7f2a3407b0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2a48150df0 tx=0x7f2a3c005f90 comp rx=0 tx=0).stop 2024-09-17T05:28:01.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 msgr2=0x7f2a4815a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a4815a7c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2a38003f30 tx=0x7f2a38001630 comp rx=0 tx=0).stop 2024-09-17T05:28:01.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 shutdown_connections 2024-09-17T05:28:01.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a34078bc0 0x7f2a3407b0d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a481567b0 0x7f2a481541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a481563a0 0x7f2a48153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 --2- 172.21.15.117:0/1472853879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30005250 0x7f2a4815a7c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 >> 172.21.15.117:0/1472853879 conn(0x7f2a4806d3d0 msgr2=0x7f2a480cb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 shutdown_connections 2024-09-17T05:28:01.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.149+0000 7f2a4e63f640 1 -- 172.21.15.117:0/1472853879 wait complete. 2024-09-17T05:28:01.162 INFO:teuthology.run_tasks:Running task kclient... 2024-09-17T05:28:01.176 INFO:tasks.kclient:Mounting kernel clients... 2024-09-17T05:28:01.176 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-17T05:28:01.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:01.176 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-09-17T05:28:01.192 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-17T05:28:01.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:01.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-17T05:28:01.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:01.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link delete ceph-brx 2024-09-17T05:28:01.360 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-09-17T05:28:01.360 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T05:28:01.360 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:01.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T05:28:01.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2256147261 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4074fb0 msgr2=0x7f5bc4151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2256147261 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4074fb0 0x7f5bc4151fd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4005fd0 tx=0x7f5bb402a5d0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2256147261 shutdown_connections 2024-09-17T05:28:01.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2256147261 >> v2:172.21.15.119:3300/0 conn(0x7f5bc4072e10 0x7f5bc40c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2256147261 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4074fb0 0x7f5bc4151fd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2256147261 >> 172.21.15.117:0/2256147261 conn(0x7f5bc406d190 msgr2=0x7f5bc406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2256147261 shutdown_connections 2024-09-17T05:28:01.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.474+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2256147261 wait complete. 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.476+0000 7f5bcaf82640 1 Processor -- start 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.476+0000 7f5bcaf82640 1 -- start start 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bcaf82640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4072e10 0x7f5bc41ef390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bcaf82640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 0x7f5bc40c0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bcaf82640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bc4142c00 0x7f5bc4143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bcaf82640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc40c8a00 con 0x7f5bc40c9800 2024-09-17T05:28:01.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bcaf82640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc4152510 con 0x7f5bc4142c00 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bcaf82640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc40c8b80 con 0x7f5bc4072e10 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bc3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 0x7f5bc40c0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bc3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 0x7f5bc40c0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42846/0 (socket says 172.21.15.117:42846) 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.477+0000 7f5bc3fff640 1 -- 172.21.15.117:0/2665883819 learned_addr learned my addr 172.21.15.117:0/2665883819 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc8d1e640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4072e10 0x7f5bc41ef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc951f640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bc4142c00 0x7f5bc4143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc3fff640 1 -- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4072e10 msgr2=0x7f5bc41ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc3fff640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4072e10 0x7f5bc41ef390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc3fff640 1 -- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bc4142c00 msgr2=0x7f5bc4143010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc3fff640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bc4142c00 0x7f5bc4143010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc3fff640 1 -- 172.21.15.117:0/2665883819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc40cbe30 con 0x7f5bc40c9800 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc8d1e640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4072e10 0x7f5bc41ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:01.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc3fff640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 0x7f5bc40c0e80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800ec20 tx=0x7f5bb800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8003ba0 con 0x7f5bc40c9800 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bb8003d40 con 0x7f5bc40c9800 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc40cc0d0 con 0x7f5bc40c9800 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8004270 con 0x7f5bc40c9800 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.478+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc40cc690 con 0x7f5bc40c9800 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.479+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc4073220 con 0x7f5bc40c9800 2024-09-17T05:28:01.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.480+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5bb800ed10 con 0x7f5bc40c9800 2024-09-17T05:28:01.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.480+0000 7f5bc17fa640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5bb0078b90 0x7f5bb007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:01.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.481+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5bb809d4c0 con 0x7f5bc40c9800 2024-09-17T05:28:01.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.481+0000 7f5bc8d1e640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5bb0078b90 0x7f5bb007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:01.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.481+0000 7f5bc8d1e640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5bb0078b90 0x7f5bb007b0a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4033000 tx=0x7f5bb4004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:01.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.485+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5bc40cbe30 con 0x7f5bc40c9800 2024-09-17T05:28:01.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.699+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5bc40c7660 con 0x7f5bc40c9800 2024-09-17T05:28:01.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.699+0000 7f5bc17fa640 1 -- 172.21.15.117:0/2665883819 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+88 (secure 0 0 0) 0x7f5bb8065650 con 0x7f5bc40c9800 2024-09-17T05:28:01.701 INFO:teuthology.orchestra.run.smithi117.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-17T05:28:01.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5bb0078b90 msgr2=0x7f5bb007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5bb0078b90 0x7f5bb007b0a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4033000 tx=0x7f5bb4004730 comp rx=0 tx=0).stop 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 msgr2=0x7f5bc40c0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 0x7f5bc40c0e80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5bb800ec20 tx=0x7f5bb800b690 comp rx=0 tx=0).stop 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 shutdown_connections 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5bb0078b90 0x7f5bb007b0a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bc4142c00 0x7f5bc4143010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc40c9800 0x7f5bc40c0e80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 --2- 172.21.15.117:0/2665883819 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5bc4072e10 0x7f5bc41ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 >> 172.21.15.117:0/2665883819 conn(0x7f5bc406d190 msgr2=0x7f5bc40c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 shutdown_connections 2024-09-17T05:28:01.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01.702+0000 7f5bcaf82640 1 -- 172.21.15.117:0/2665883819 wait complete. 2024-09-17T05:28:01.714 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi117.front.sepia.ceph.com 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T05:28:01.715 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T05:28:01.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:01.715 DEBUG:teuthology.orchestra.run.smithi117:> ip addr 2024-09-17T05:28:01.731 INFO:teuthology.orchestra.run.smithi117.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 ::1/128 scope host 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8a brd ff:ff:ff:ff:ff:ff 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f0 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8b brd ff:ff:ff:ff:ff:ff 2024-09-17T05:28:01.732 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f1 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:62 brd ff:ff:ff:ff:ff:ff 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:63 brd ff:ff:ff:ff:ff:ff 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: inet 172.21.15.117/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft 42757sec preferred_lft 42757sec 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 fe80::ec4:7aff:fe8f:4463/64 scope link noprefixroute 2024-09-17T05:28:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-09-17T05:28:01.733 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add name ceph-brx type bridge 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr flush dev ceph-brx 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set ceph-brx up 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-17T05:28:01.734 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:28:01.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:28:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:01.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:01.871 DEBUG:teuthology.orchestra.run.smithi117:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T05:28:01.894 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-09-17T05:28:01.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:01.895 DEBUG:teuthology.orchestra.run.smithi117:> route 2024-09-17T05:28:01.953 INFO:teuthology.orchestra.run.smithi117.stdout:Kernel IP routing table 2024-09-17T05:28:01.953 INFO:teuthology.orchestra.run.smithi117.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T05:28:01.953 INFO:teuthology.orchestra.run.smithi117.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T05:28:01.953 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T05:28:01.953 INFO:teuthology.orchestra.run.smithi117.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T05:28:01.954 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:28:01.954 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:28:01.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T05:28:01.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T05:28:01.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T05:28:01.954 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:28:02.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:28:02.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:02.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:02.081 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-09-17T05:28:02.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:02.097 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list-id 2024-09-17T05:28:02.153 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:28:02.153 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:28:02.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T05:28:02.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-17T05:28:02.153 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:28:02.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:28:02.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:02.269 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-17T05:28:02.270 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:28:02.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:28:02.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:02.449 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T05:28:02.449 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-09-17T05:28:02.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set brx.0 up 2024-09-17T05:28:02.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set dev brx.0 master ceph-brx 2024-09-17T05:28:02.449 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-09-17T05:28:02.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:28:02.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:02.525 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T05:28:02.526 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:02.550 INFO:teuthology.orchestra.run.smithi117.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T05:28:02.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T05:28:02.551 DEBUG:teuthology.orchestra.run.smithi117:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:02.605 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T05:28:02.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:02.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T05:28:02.701 INFO:teuthology.orchestra.run.smithi117.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T05:28:02.701 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T05:28:02.701 INFO:teuthology.orchestra.run.smithi117.stdout:invalid new device string format 2024-09-17T05:28:02.839 INFO:teuthology.orchestra.run.smithi117.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T05:28:02.839 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T05:28:02.839 INFO:teuthology.orchestra.run.smithi117.stdout:invalid new device string format 2024-09-17T05:28:02.839 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: resolved to: "172.21.15.117:3300,172.21.15.119:3300,172.21.15.119:3301" 2024-09-17T05:28:02.839 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T05:28:02.845 INFO:teuthology.orchestra.run.smithi117.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T05:28:02.845 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T05:28:02.846 INFO:teuthology.orchestra.run.smithi117.stdout:invalid new device string format 2024-09-17T05:28:02.846 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: resolved to: "172.21.15.117:3300,172.21.15.119:3300,172.21.15.119:3301" 2024-09-17T05:28:02.846 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T05:28:02.846 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: resolved to: "172.21.15.117:6789,172.21.15.119:6789,172.21.15.119:6790" 2024-09-17T05:28:02.846 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.117:6789,172.21.15.119:6789,172.21.15.119:6790:/ 2024-09-17T05:28:02.846 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=8527add3-c46f-4f7f-86e8-5cd706a0043e" will pass to kernel 2024-09-17T05:28:02.847 INFO:teuthology.orchestra.run.smithi117.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T05:28:02.847 INFO:teuthology.orchestra.run.smithi117.stdout: You just mounted a file system that supports labels which does not 2024-09-17T05:28:02.847 INFO:teuthology.orchestra.run.smithi117.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T05:28:02.847 INFO:teuthology.orchestra.run.smithi117.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T05:28:02.847 INFO:teuthology.orchestra.run.smithi117.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T05:28:02.848 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T05:28:02.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:28:02.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:02.873 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T05:28:02.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T05:28:02.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:02.970 INFO:teuthology.orchestra.run.smithi117.stdout:client4700 2024-09-17T05:28:02.970 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:28:02.971 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:28:02.987 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T05:28:03.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T05:28:03.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:03.083 INFO:teuthology.orchestra.run.smithi117.stdout:client4700 2024-09-17T05:28:03.084 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T05:28:03.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T05:28:03.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:client4700 2024-09-17T05:28:03.141 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8527add3-c46f-4f7f-86e8-5cd706a0043e.client4700/status') 2024-09-17T05:28:03.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T05:28:03.199 INFO:teuthology.orchestra.run.smithi117.stdout:instance: client.4700 (3)172.21.15.117:0/3288242122 2024-09-17T05:28:03.199 INFO:teuthology.orchestra.run.smithi117.stdout:blocklisted: false 2024-09-17T05:28:03.199 INFO:teuthology.orchestra.run.smithi117.stderr:0+1 records in 2024-09-17T05:28:03.200 INFO:teuthology.orchestra.run.smithi117.stderr:0+1 records out 2024-09-17T05:28:03.200 INFO:teuthology.orchestra.run.smithi117.stderr:71 bytes copied, 2.1515e-05 s, 3.3 MB/s 2024-09-17T05:28:03.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T05:28:03.207 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-09-17T05:28:03.220 INFO:tasks.mds_thrash:mds thrasher using random seed: 1726550883 2024-09-17T05:28:03.221 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/2736221339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf41530e0 msgr2=0x7f6bf40ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/2736221339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf41530e0 0x7f6bf40ce590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6bec005fd0 tx=0x7f6bec02aa10 comp rx=0 tx=0).stop 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/2736221339 shutdown_connections 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/2736221339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf41530e0 0x7f6bf40ce590 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/2736221339 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6bf40c9e60 0x7f6bf40c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/2736221339 >> 172.21.15.117:0/2736221339 conn(0x7f6bf406cda0 msgr2=0x7f6bf406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:03.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/2736221339 shutdown_connections 2024-09-17T05:28:03.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.335+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/2736221339 wait complete. 2024-09-17T05:28:03.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.337+0000 7f6bfa0a9640 1 Processor -- start 2024-09-17T05:28:03.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.337+0000 7f6bfa0a9640 1 -- start start 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bfa0a9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 0x7f6bf41f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bfa0a9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6bf41530e0 0x7f6bf4149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bfa0a9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf40cceb0 0x7f6bf414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bfa0a9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf40c3a80 con 0x7f6bf40c9e60 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bfa0a9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf4159540 con 0x7f6bf40cceb0 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bfa0a9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf40cad80 con 0x7f6bf41530e0 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.338+0000 7f6bf37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 0x7f6bf41f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 0x7f6bf41f4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42874/0 (socket says 172.21.15.117:42874) 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 -- 172.21.15.117:0/1924074526 learned_addr learned my addr 172.21.15.117:0/1924074526 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf3fff640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf40cceb0 0x7f6bf414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf2ffd640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6bf41530e0 0x7f6bf4149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 -- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6bf41530e0 msgr2=0x7f6bf4149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6bf41530e0 0x7f6bf4149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 -- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf40cceb0 msgr2=0x7f6bf414e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf40cceb0 0x7f6bf414e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 -- 172.21.15.117:0/1924074526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf40c73d0 con 0x7f6bf40c9e60 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bf37fe640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 0x7f6bf41f4610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6be400c400 tx=0x7f6be400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4017070 con 0x7f6bf40c9e60 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6be40053c0 con 0x7f6bf40c9e60 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4003850 con 0x7f6bf40c9e60 2024-09-17T05:28:03.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf40c7b20 con 0x7f6bf40c9e60 2024-09-17T05:28:03.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.339+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf40c80e0 con 0x7f6bf40c9e60 2024-09-17T05:28:03.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.340+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf41535d0 con 0x7f6bf40c9e60 2024-09-17T05:28:03.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.341+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6be401a070 con 0x7f6bf40c9e60 2024-09-17T05:28:03.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.342+0000 7f6bebfff640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bd4078bc0 0x7f6bd407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.342+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6be409a2e0 con 0x7f6bf40c9e60 2024-09-17T05:28:03.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.342+0000 7f6bf2ffd640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bd4078bc0 0x7f6bd407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.343+0000 7f6bf2ffd640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bd4078bc0 0x7f6bd407b0d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4151fa0 tx=0x7f6bdc00b2e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:03.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.346+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6be4061e20 con 0x7f6bf40c9e60 2024-09-17T05:28:03.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.530+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bf40ca950 con 0x7f6bf40c9e60 2024-09-17T05:28:03.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.532+0000 7f6bebfff640 1 -- 172.21.15.117:0/1924074526 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18674 (secure 0 0 0) 0x7f6be4066850 con 0x7f6bf40c9e60 2024-09-17T05:28:03.534 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:03.534 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:27:31.028778+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bd4078bc0 msgr2=0x7f6bd407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bd4078bc0 0x7f6bd407b0d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4151fa0 tx=0x7f6bdc00b2e0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 msgr2=0x7f6bf41f4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 0x7f6bf41f4610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6be400c400 tx=0x7f6be400c8c0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 shutdown_connections 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bd4078bc0 0x7f6bd407b0d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf40cceb0 0x7f6bf414e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6bf41530e0 0x7f6bf4149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 --2- 172.21.15.117:0/1924074526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bf40c9e60 0x7f6bf41f4610 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 >> 172.21.15.117:0/1924074526 conn(0x7f6bf406cda0 msgr2=0x7f6bf4072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:03.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 shutdown_connections 2024-09-17T05:28:03.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.534+0000 7f6bfa0a9640 1 -- 172.21.15.117:0/1924074526 wait complete. 2024-09-17T05:28:03.548 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T05:27:13.363384+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-17T05:27:23.869446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-17T05:27:24.963547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-17T05:27:28.659816+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-17T05:28:03.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T05:28:03.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 -- 172.21.15.117:0/4028133759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa8074df0 msgr2=0x7f3fa815b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 --2- 172.21.15.117:0/4028133759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa8074df0 0x7f3fa815b450 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8072350 tx=0x7f3f9c02a560 comp rx=0 tx=0).stop 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 -- 172.21.15.117:0/4028133759 shutdown_connections 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 --2- 172.21.15.117:0/4028133759 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3fa80c67d0 0x7f3fa80c3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 --2- 172.21.15.117:0/4028133759 >> v2:172.21.15.119:3300/0 conn(0x7f3fa80c5e80 0x7f3fa80c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 --2- 172.21.15.117:0/4028133759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa8074df0 0x7f3fa815b450 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 -- 172.21.15.117:0/4028133759 >> 172.21.15.117:0/4028133759 conn(0x7f3fa80ccbe0 msgr2=0x7f3fa80c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 -- 172.21.15.117:0/4028133759 shutdown_connections 2024-09-17T05:28:03.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.664+0000 7f3faefc3640 1 -- 172.21.15.117:0/4028133759 wait complete. 2024-09-17T05:28:03.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.666+0000 7f3faefc3640 1 Processor -- start 2024-09-17T05:28:03.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.666+0000 7f3faefc3640 1 -- start start 2024-09-17T05:28:03.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3faefc3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3fa8074df0 0x7f3fa81e3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3faefc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 0x7f3fa814f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3faefc3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa80c67d0 0x7f3fa814d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3faefc3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa81586e0 con 0x7f3fa80c5e80 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3faefc3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa8158400 con 0x7f3fa80c67d0 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3faefc3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa80c59f0 con 0x7f3fa8074df0 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3fa7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 0x7f3fa814f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3fa7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 0x7f3fa814f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42904/0 (socket says 172.21.15.117:42904) 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3fa7fff640 1 -- 172.21.15.117:0/1126673158 learned_addr learned my addr 172.21.15.117:0/1126673158 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3facd5f640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3fa8074df0 0x7f3fa81e3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.667+0000 7f3fad560640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa80c67d0 0x7f3fa814d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa7fff640 1 -- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3fa8074df0 msgr2=0x7f3fa81e3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa7fff640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3fa8074df0 0x7f3fa81e3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa7fff640 1 -- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa80c67d0 msgr2=0x7f3fa814d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa7fff640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa80c67d0 0x7f3fa814d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa7fff640 1 -- 172.21.15.117:0/1126673158 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa81e2b10 con 0x7f3fa80c5e80 2024-09-17T05:28:03.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa7fff640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 0x7f3fa814f030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400d680 tx=0x7f3f9400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:03.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94017070 con 0x7f3fa80c5e80 2024-09-17T05:28:03.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f94004310 con 0x7f3fa80c5e80 2024-09-17T05:28:03.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa814dc90 con 0x7f3fa80c5e80 2024-09-17T05:28:03.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94005280 con 0x7f3fa80c5e80 2024-09-17T05:28:03.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.668+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa814fcb0 con 0x7f3fa80c5e80 2024-09-17T05:28:03.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.669+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa80c9760 con 0x7f3fa80c5e80 2024-09-17T05:28:03.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.670+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f94005420 con 0x7f3fa80c5e80 2024-09-17T05:28:03.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.670+0000 7f3fa57fa640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f8c078bc0 0x7f3f8c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.670+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3f9409b3d0 con 0x7f3fa80c5e80 2024-09-17T05:28:03.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.670+0000 7f3facd5f640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f8c078bc0 0x7f3f8c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.671+0000 7f3facd5f640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f8c078bc0 0x7f3f8c07b0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c033000 tx=0x7f3f9c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:03.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.675+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3f94020020 con 0x7f3fa80c5e80 2024-09-17T05:28:03.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.856+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3fa80738e0 con 0x7f3fa80c5e80 2024-09-17T05:28:03.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.856+0000 7f3fa57fa640 1 -- 172.21.15.117:0/1126673158 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f3f94063560 con 0x7f3fa80c5e80 2024-09-17T05:28:03.858 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.858+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f8c078bc0 msgr2=0x7f3f8c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.858+0000 7f3faefc3640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f8c078bc0 0x7f3f8c07b0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c033000 tx=0x7f3f9c004730 comp rx=0 tx=0).stop 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.858+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 msgr2=0x7f3fa814f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.858+0000 7f3faefc3640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 0x7f3fa814f030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400d680 tx=0x7f3f9400db40 comp rx=0 tx=0).stop 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 shutdown_connections 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3f8c078bc0 0x7f3f8c07b0d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa80c67d0 0x7f3fa814d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa80c5e80 0x7f3fa814f030 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 --2- 172.21.15.117:0/1126673158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3fa8074df0 0x7f3fa81e3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 >> 172.21.15.117:0/1126673158 conn(0x7f3fa80ccbe0 msgr2=0x7f3fa806cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:03.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 shutdown_connections 2024-09-17T05:28:03.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.859+0000 7f3faefc3640 1 -- 172.21.15.117:0/1126673158 wait complete. 2024-09-17T05:28:03.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 -- 172.21.15.117:0/1275511885 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb5180b56c0 msgr2=0x7fb518165c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 --2- 172.21.15.117:0/1275511885 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb5180b56c0 0x7fb518165c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb510005fd0 tx=0x7fb51002a940 comp rx=0 tx=0).stop 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 -- 172.21.15.117:0/1275511885 shutdown_connections 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 --2- 172.21.15.117:0/1275511885 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb5180b56c0 0x7fb518165c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 --2- 172.21.15.117:0/1275511885 >> v2:172.21.15.119:3300/0 conn(0x7fb51815bd70 0x7fb5180ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 -- 172.21.15.117:0/1275511885 >> 172.21.15.117:0/1275511885 conn(0x7fb518074d80 msgr2=0x7fb5180d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 -- 172.21.15.117:0/1275511885 shutdown_connections 2024-09-17T05:28:03.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.987+0000 7fb51d04a640 1 -- 172.21.15.117:0/1275511885 wait complete. 2024-09-17T05:28:03.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.989+0000 7fb51d04a640 1 Processor -- start 2024-09-17T05:28:03.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.989+0000 7fb51d04a640 1 -- start start 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.990+0000 7fb51d04a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5180b52b0 0x7fb51815e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.990+0000 7fb51d04a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 0x7fb518154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.990+0000 7fb51d04a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb51815bd70 0x7fb518155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.990+0000 7fb51d04a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb5180cacb0 con 0x7fb5180b56c0 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.990+0000 7fb51d04a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb5180cd0d0 con 0x7fb5180b52b0 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.990+0000 7fb51d04a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb5181558b0 con 0x7fb51815bd70 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 0x7fb518154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 0x7fb518154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40390/0 (socket says 172.21.15.117:40390) 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 -- 172.21.15.117:0/2829972268 learned_addr learned my addr 172.21.15.117:0/2829972268 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:03.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb516d9d640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5180b52b0 0x7fb51815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51759e640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb51815bd70 0x7fb518155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 -- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb51815bd70 msgr2=0x7fb518155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb51815bd70 0x7fb518155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 -- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5180b52b0 msgr2=0x7fb51815e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5180b52b0 0x7fb51815e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 -- 172.21.15.117:0/2829972268 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5180c8a00 con 0x7fb5180b56c0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51759e640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb51815bd70 0x7fb518155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51659c640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 0x7fb518154c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c3a0 tx=0x7fb50c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c011790 con 0x7fb5180b56c0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb50c017070 con 0x7fb5180b56c0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb518158790 con 0x7fb5180b56c0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c011cc0 con 0x7fb5180b56c0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.991+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51815c6c0 con 0x7fb5180b56c0 2024-09-17T05:28:03.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.992+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5181f5a50 con 0x7fb5180b56c0 2024-09-17T05:28:03.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.993+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb50c01c460 con 0x7fb5180b56c0 2024-09-17T05:28:03.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.994+0000 7fb4ff7fe640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb504078b90 0x7fb50407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:03.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.994+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb50c09ad30 con 0x7fb5180b56c0 2024-09-17T05:28:03.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.994+0000 7fb516d9d640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb504078b90 0x7fb50407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:03.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:03.994+0000 7fb516d9d640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb504078b90 0x7fb50407b0a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb508009ac0 tx=0x7fb508005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.003+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb50c062f50 con 0x7fb5180b56c0 2024-09-17T05:28:04.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.184+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb51806af90 con 0x7fb5180b56c0 2024-09-17T05:28:04.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.185+0000 7fb4ff7fe640 1 -- 172.21.15.117:0/2829972268 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fb51806af90 con 0x7fb5180b56c0 2024-09-17T05:28:04.186 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-09-17T05:28:04.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb504078b90 msgr2=0x7fb50407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb504078b90 0x7fb50407b0a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb508009ac0 tx=0x7fb508005f90 comp rx=0 tx=0).stop 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 msgr2=0x7fb518154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 0x7fb518154c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c3a0 tx=0x7fb50c00c860 comp rx=0 tx=0).stop 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 shutdown_connections 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb504078b90 0x7fb50407b0a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb51815bd70 0x7fb518155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5180b56c0 0x7fb518154c40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 --2- 172.21.15.117:0/2829972268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5180b52b0 0x7fb51815e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 >> 172.21.15.117:0/2829972268 conn(0x7fb518074d80 msgr2=0x7fb5180cc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 shutdown_connections 2024-09-17T05:28:04.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.187+0000 7fb51d04a640 1 -- 172.21.15.117:0/2829972268 wait complete. 2024-09-17T05:28:04.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-17T05:28:04.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 -- 172.21.15.117:0/1845954457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d740c67a0 msgr2=0x7f0d74152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 --2- 172.21.15.117:0/1845954457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d740c67a0 0x7f0d74152210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d740ce5b0 tx=0x7f0d6c02a4a0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 -- 172.21.15.117:0/1845954457 shutdown_connections 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 --2- 172.21.15.117:0/1845954457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d740c67a0 0x7f0d74152210 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 -- 172.21.15.117:0/1845954457 >> 172.21.15.117:0/1845954457 conn(0x7f0d7406e590 msgr2=0x7f0d740cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 -- 172.21.15.117:0/1845954457 shutdown_connections 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.317+0000 7f0d7b640640 1 -- 172.21.15.117:0/1845954457 wait complete. 2024-09-17T05:28:04.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.318+0000 7f0d7b640640 1 Processor -- start 2024-09-17T05:28:04.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.318+0000 7f0d7b640640 1 -- start start 2024-09-17T05:28:04.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d7b640640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d74071850 0x7f0d741e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d7b640640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d740c5660 0x7f0d74146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d7b640640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 0x7f0d741474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d7b640640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0d740c7ff0 con 0x7f0d740c67a0 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d7b640640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0d740ce0d0 con 0x7f0d74071850 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d7b640640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0d740c83a0 con 0x7f0d740c5660 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 0x7f0d741474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 0x7f0d741474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40400/0 (socket says 172.21.15.117:40400) 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 -- 172.21.15.117:0/3824098723 learned_addr learned my addr 172.21.15.117:0/3824098723 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d793dc640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d74071850 0x7f0d741e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d78bdb640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d740c5660 0x7f0d74146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 -- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d740c5660 msgr2=0x7f0d74146fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d740c5660 0x7f0d74146fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 -- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d74071850 msgr2=0x7f0d741e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d74071850 0x7f0d741e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 -- 172.21.15.117:0/3824098723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7415ad40 con 0x7f0d740c67a0 2024-09-17T05:28:04.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d79bdd640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 0x7f0d741474e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400cbb0 tx=0x7f0d6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.320+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64016020 con 0x7f0d740c67a0 2024-09-17T05:28:04.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.321+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7415b020 con 0x7f0d740c67a0 2024-09-17T05:28:04.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.321+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d6400ae30 con 0x7f0d740c67a0 2024-09-17T05:28:04.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.321+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64012680 con 0x7f0d740c67a0 2024-09-17T05:28:04.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.321+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d741e2760 con 0x7f0d740c67a0 2024-09-17T05:28:04.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.322+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d740c7940 con 0x7f0d740c67a0 2024-09-17T05:28:04.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.322+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d64019020 con 0x7f0d740c67a0 2024-09-17T05:28:04.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.323+0000 7f0d71ffb640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d60078b90 0x7f0d6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.323+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0d6409b760 con 0x7f0d740c67a0 2024-09-17T05:28:04.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.323+0000 7f0d793dc640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d60078b90 0x7f0d6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.324+0000 7f0d793dc640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d60078b90 0x7f0d6007b0a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c034000 tx=0x7f0d6c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.327+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d64020020 con 0x7f0d740c67a0 2024-09-17T05:28:04.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.502+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0d740c8d00 con 0x7f0d740c67a0 2024-09-17T05:28:04.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.503+0000 7f0d71ffb640 1 -- 172.21.15.117:0/3824098723 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f0d64063220 con 0x7f0d740c67a0 2024-09-17T05:28:04.504 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-09-17T05:28:04.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d60078b90 msgr2=0x7f0d6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d60078b90 0x7f0d6007b0a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c034000 tx=0x7f0d6c004730 comp rx=0 tx=0).stop 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 msgr2=0x7f0d741474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 0x7f0d741474e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400cbb0 tx=0x7f0d6400a430 comp rx=0 tx=0).stop 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 shutdown_connections 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d60078b90 0x7f0d6007b0a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d740c67a0 0x7f0d741474e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d740c5660 0x7f0d74146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 --2- 172.21.15.117:0/3824098723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d74071850 0x7f0d741e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 >> 172.21.15.117:0/3824098723 conn(0x7f0d7406e590 msgr2=0x7f0d740c6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 shutdown_connections 2024-09-17T05:28:04.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.505+0000 7f0d7b640640 1 -- 172.21.15.117:0/3824098723 wait complete. 2024-09-17T05:28:04.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-17T05:28:04.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.630+0000 7f84e99da640 1 -- 172.21.15.117:0/751593605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 msgr2=0x7f84d0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.630+0000 7f84e99da640 1 --2- 172.21.15.117:0/751593605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84d0003ea0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f84d40016c0 tx=0x7f84d0005830 comp rx=0 tx=0).stop 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.631+0000 7f84e37fe640 1 -- 172.21.15.117:0/751593605 reap_dead start 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.631+0000 7f84e99da640 1 -- 172.21.15.117:0/751593605 shutdown_connections 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.631+0000 7f84e99da640 1 -- 172.21.15.117:0/751593605 >> 172.21.15.117:0/751593605 conn(0x7f84e406e590 msgr2=0x7f84e40cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.631+0000 7f84e99da640 1 -- 172.21.15.117:0/751593605 shutdown_connections 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.631+0000 7f84e99da640 1 -- 172.21.15.117:0/751593605 wait complete. 2024-09-17T05:28:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.632+0000 7f84e99da640 1 Processor -- start 2024-09-17T05:28:04.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.633+0000 7f84e99da640 1 -- start start 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e99da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84e4153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e99da640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f84e4155900 0x7f84e414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e99da640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84e41547e0 0x7f84e41f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e99da640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f84e40c7ff0 con 0x7f84d0003a30 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e99da640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f84e40ce0d0 con 0x7f84e41547e0 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e99da640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f84e40c83a0 con 0x7f84e4155900 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84e4153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84e4153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40418/0 (socket says 172.21.15.117:40418) 2024-09-17T05:28:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 -- 172.21.15.117:0/2531875295 learned_addr learned my addr 172.21.15.117:0/2531875295 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e37fe640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84e41547e0 0x7f84e41f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84da7fc640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f84e4155900 0x7f84e414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 -- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f84e4155900 msgr2=0x7f84e414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f84e4155900 0x7f84e414ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 -- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84e41547e0 msgr2=0x7f84e41f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84e41547e0 0x7f84e41f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 -- 172.21.15.117:0/2531875295 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e4154390 con 0x7f84d0003a30 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84da7fc640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f84e4155900 0x7f84e414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e37fe640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84e41547e0 0x7f84e41f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.634+0000 7f84e2ffd640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84e4153980 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f84d402a930 tx=0x7f84d4006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.635+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4038070 con 0x7f84d0003a30 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.635+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84d4003af0 con 0x7f84d0003a30 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.635+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d403d460 con 0x7f84d0003a30 2024-09-17T05:28:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.635+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e4153ec0 con 0x7f84d0003a30 2024-09-17T05:28:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.635+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e41551d0 con 0x7f84d0003a30 2024-09-17T05:28:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.636+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e40c7940 con 0x7f84d0003a30 2024-09-17T05:28:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.636+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f84d4030070 con 0x7f84d0003a30 2024-09-17T05:28:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.637+0000 7f84e0ff9640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f84c4078b90 0x7f84c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.637+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f84d40baa60 con 0x7f84d0003a30 2024-09-17T05:28:04.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.637+0000 7f84da7fc640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f84c4078b90 0x7f84c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.638+0000 7f84da7fc640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f84c4078b90 0x7f84c407b0a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f84cc001eb0 tx=0x7f84cc001110 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.641+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f84e4154390 con 0x7f84d0003a30 2024-09-17T05:28:04.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.814+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f84e40c8d00 con 0x7f84d0003a30 2024-09-17T05:28:04.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.814+0000 7f84e0ff9640 1 -- 172.21.15.117:0/2531875295 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f84d4082ae0 con 0x7f84d0003a30 2024-09-17T05:28:04.816 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-09-17T05:28:04.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f84c4078b90 msgr2=0x7f84c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f84c4078b90 0x7f84c407b0a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f84cc001eb0 tx=0x7f84cc001110 comp rx=0 tx=0).stop 2024-09-17T05:28:04.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 msgr2=0x7f84e4153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84e4153980 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f84d402a930 tx=0x7f84d4006360 comp rx=0 tx=0).stop 2024-09-17T05:28:04.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 shutdown_connections 2024-09-17T05:28:04.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f84c4078b90 0x7f84c407b0a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84e41547e0 0x7f84e41f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f84e4155900 0x7f84e414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 --2- 172.21.15.117:0/2531875295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84d0003a30 0x7f84e4153980 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 >> 172.21.15.117:0/2531875295 conn(0x7f84e406e590 msgr2=0x7f84e4158560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:04.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 shutdown_connections 2024-09-17T05:28:04.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.816+0000 7f84e99da640 1 -- 172.21.15.117:0/2531875295 wait complete. 2024-09-17T05:28:04.829 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-09-17T05:28:04.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:28:04.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d03d20640 1 -- 172.21.15.117:0/4009648045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 msgr2=0x7f0ce40051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d03d20640 1 --2- 172.21.15.117:0/4009648045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0ce40051f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0c8b00 tx=0x7f0ce4005830 comp rx=0 tx=0).stop 2024-09-17T05:28:04.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d022bd640 1 -- 172.21.15.117:0/4009648045 reap_dead start 2024-09-17T05:28:04.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d03d20640 1 -- 172.21.15.117:0/4009648045 shutdown_connections 2024-09-17T05:28:04.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d03d20640 1 -- 172.21.15.117:0/4009648045 >> 172.21.15.117:0/4009648045 conn(0x7f0cfc06d190 msgr2=0x7f0cfc06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:04.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d03d20640 1 -- 172.21.15.117:0/4009648045 shutdown_connections 2024-09-17T05:28:04.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.939+0000 7f0d03d20640 1 -- 172.21.15.117:0/4009648045 wait complete. 2024-09-17T05:28:04.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.941+0000 7f0d03d20640 1 Processor -- start 2024-09-17T05:28:04.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.941+0000 7f0d03d20640 1 -- start start 2024-09-17T05:28:04.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.942+0000 7f0d03d20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0cfc14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.942+0000 7f0d03d20640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cfc0bf9c0 0x7f0cfc0be780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.942+0000 7f0d03d20640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0cfc154170 0x7f0cfc141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.942+0000 7f0d03d20640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc075100 con 0x7f0ce4004d40 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.942+0000 7f0d03d20640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc074c00 con 0x7f0cfc0bf9c0 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.942+0000 7f0d03d20640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0cfc0c99d0 con 0x7f0cfc154170 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0cfc14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0cfc14dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40436/0 (socket says 172.21.15.117:40436) 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 -- 172.21.15.117:0/1855899826 learned_addr learned my addr 172.21.15.117:0/1855899826 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d022bd640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0cfc154170 0x7f0cfc141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d012bb640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cfc0bf9c0 0x7f0cfc0be780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 -- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0cfc154170 msgr2=0x7f0cfc141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0cfc154170 0x7f0cfc141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 -- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cfc0bf9c0 msgr2=0x7f0cfc0be780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cfc0bf9c0 0x7f0cfc0be780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 -- 172.21.15.117:0/1855899826 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc0bdb30 con 0x7f0ce4004d40 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d022bd640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0cfc154170 0x7f0cfc141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d01abc640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0cfc14dad0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0cec011d60 tx=0x7f0cec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec003d80 con 0x7f0ce4004d40 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cec011e50 con 0x7f0ce4004d40 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cfc0bd570 con 0x7f0ce4004d40 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec004d40 con 0x7f0ce4004d40 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.943+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cfc0bf1c0 con 0x7f0ce4004d40 2024-09-17T05:28:04.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.944+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cfc0c6630 con 0x7f0ce4004d40 2024-09-17T05:28:04.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.945+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0cec004050 con 0x7f0ce4004d40 2024-09-17T05:28:04.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.945+0000 7f0cfa7fc640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0ce8078bc0 0x7f0ce807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:04.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.946+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0cec09c0e0 con 0x7f0ce4004d40 2024-09-17T05:28:04.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.946+0000 7f0d012bb640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0ce8078bc0 0x7f0ce807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:04.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.946+0000 7f0d012bb640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0ce8078bc0 0x7f0ce807b0d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc14ac70 tx=0x7f0cf0006b70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:04.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:04.950+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0cec064270 con 0x7f0ce4004d40 2024-09-17T05:28:05.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.160+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cfc0cd350 con 0x7f0ce4004d40 2024-09-17T05:28:05.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.161+0000 7f0cfa7fc640 1 -- 172.21.15.117:0/1855899826 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7f0cec063b70 con 0x7f0ce4004d40 2024-09-17T05:28:05.162 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:05.162 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:28:05.163 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:28:05.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0ce8078bc0 msgr2=0x7f0ce807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0ce8078bc0 0x7f0ce807b0d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc14ac70 tx=0x7f0cf0006b70 comp rx=0 tx=0).stop 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 msgr2=0x7f0cfc14dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0cfc14dad0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0cec011d60 tx=0x7f0cec00f790 comp rx=0 tx=0).stop 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 shutdown_connections 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0ce8078bc0 0x7f0ce807b0d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0cfc154170 0x7f0cfc141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cfc0bf9c0 0x7f0cfc0be780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 --2- 172.21.15.117:0/1855899826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ce4004d40 0x7f0cfc14dad0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 >> 172.21.15.117:0/1855899826 conn(0x7f0cfc06d190 msgr2=0x7f0cfc151e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 shutdown_connections 2024-09-17T05:28:05.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.163+0000 7f0d03d20640 1 -- 172.21.15.117:0/1855899826 wait complete. 2024-09-17T05:28:05.175 INFO:tasks.mds_thrash:Ready to start thrashing 2024-09-17T05:28:05.175 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-09-17T05:28:05.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:28:05.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.285+0000 7febd3af9640 1 -- 172.21.15.117:0/3487932779 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c6fa0 msgr2=0x7febcc075080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.285+0000 7febd3af9640 1 --2- 172.21.15.117:0/3487932779 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c6fa0 0x7febcc075080 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7febbc009860 tx=0x7febbc02a530 comp rx=0 tx=0).stop 2024-09-17T05:28:05.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.286+0000 7febd3af9640 1 -- 172.21.15.117:0/3487932779 shutdown_connections 2024-09-17T05:28:05.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.286+0000 7febd3af9640 1 --2- 172.21.15.117:0/3487932779 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c6fa0 0x7febcc075080 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.286+0000 7febd3af9640 1 -- 172.21.15.117:0/3487932779 >> 172.21.15.117:0/3487932779 conn(0x7febcc06cda0 msgr2=0x7febcc06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:05.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.286+0000 7febd3af9640 1 -- 172.21.15.117:0/3487932779 shutdown_connections 2024-09-17T05:28:05.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.286+0000 7febd3af9640 1 -- 172.21.15.117:0/3487932779 wait complete. 2024-09-17T05:28:05.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.287+0000 7febd3af9640 1 Processor -- start 2024-09-17T05:28:05.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.287+0000 7febd3af9640 1 -- start start 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.288+0000 7febd3af9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 0x7febcc14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.288+0000 7febd3af9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c8070 0x7febcc1554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.288+0000 7febd3af9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7febcc1534a0 0x7febcc152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd3af9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7febcc0c7b50 con 0x7febcc0c6fa0 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd3af9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7febcc0c7900 con 0x7febcc1534a0 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd3af9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7febcc070530 con 0x7febcc0c8070 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 0x7febcc14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 0x7febcc14e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40448/0 (socket says 172.21.15.117:40448) 2024-09-17T05:28:05.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 -- 172.21.15.117:0/4255615459 learned_addr learned my addr 172.21.15.117:0/4255615459 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd2096640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7febcc1534a0 0x7febcc152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1094640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c8070 0x7febcc1554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 -- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c8070 msgr2=0x7febcc1554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c8070 0x7febcc1554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 -- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7febcc1534a0 msgr2=0x7febcc152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7febcc1534a0 0x7febcc152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 -- 172.21.15.117:0/4255615459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc156a10 con 0x7febcc0c6fa0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd1895640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 0x7febcc14e800 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7febc800c460 tx=0x7febc800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc80052d0 con 0x7febcc0c6fa0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febc8005490 con 0x7febcc0c6fa0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febcc1f5fe0 con 0x7febcc0c6fa0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8012870 con 0x7febcc0c6fa0 2024-09-17T05:28:05.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.289+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febcc14f150 con 0x7febcc0c6fa0 2024-09-17T05:28:05.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.290+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febcc076270 con 0x7febcc0c6fa0 2024-09-17T05:28:05.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.291+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7febc8012a10 con 0x7febcc0c6fa0 2024-09-17T05:28:05.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.291+0000 7febc27fc640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7febb0078bc0 0x7febb007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.292+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7febc809abe0 con 0x7febcc0c6fa0 2024-09-17T05:28:05.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.292+0000 7febd1094640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7febb0078bc0 0x7febb007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.292+0000 7febd1094640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7febb0078bc0 0x7febb007b0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7febbc034000 tx=0x7febbc004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:05.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.296+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7febcc156a10 con 0x7febcc0c6fa0 2024-09-17T05:28:05.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.470+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7febcc153c70 con 0x7febb0078bc0 2024-09-17T05:28:05.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.476+0000 7febc27fc640 1 -- 172.21.15.117:0/4255615459 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332665 (secure 0 0 0) 0x7febcc153c70 con 0x7febb0078bc0 2024-09-17T05:28:05.481 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:05.483 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7febb0078bc0 msgr2=0x7febb007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7febb0078bc0 0x7febb007b0d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7febbc034000 tx=0x7febbc004730 comp rx=0 tx=0).stop 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 msgr2=0x7febcc14e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 0x7febcc14e800 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7febc800c460 tx=0x7febc800c920 comp rx=0 tx=0).stop 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 shutdown_connections 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7febb0078bc0 0x7febb007b0d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7febcc1534a0 0x7febcc152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7febcc0c8070 0x7febcc1554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 --2- 172.21.15.117:0/4255615459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febcc0c6fa0 0x7febcc14e800 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 >> 172.21.15.117:0/4255615459 conn(0x7febcc06cda0 msgr2=0x7febcc1e6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:05.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 shutdown_connections 2024-09-17T05:28:05.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.479+0000 7febd3af9640 1 -- 172.21.15.117:0/4255615459 wait complete. 2024-09-17T05:28:05.492 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-17T05:28:05.359077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598076,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":230,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218496,"kb_used_data":2304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756224,"statfs":{"total":773094113280,"available":772870373376,"internally_reserved":0,"allocated":2359296,"data_stored":2012400,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1662,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003165"},"pg_stats":[{"pgid":"2.1f","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770490+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:54.770490+0000","last_peered":"2024-09-17T05:27:54.770490+0000","last_clean":"2024-09-17T05:27:54.770490+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:54.770490+0000","last_undegraded":"2024-09-17T05:27:54.770490+0000","last_fullsized":"2024-09-17T05:27:54.770490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688890+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688890+0000","last_peered":"2024-09-17T05:27:31.688890+0000","last_clean":"2024-09-17T05:27:31.688890+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688890+0000","last_undegraded":"2024-09-17T05:27:31.688890+0000","last_fullsized":"2024-09-17T05:27:31.688890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.769713+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:54.769713+0000","last_peered":"2024-09-17T05:27:54.769713+0000","last_clean":"2024-09-17T05:27:54.769713+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:54.769713+0000","last_undegraded":"2024-09-17T05:27:54.769713+0000","last_fullsized":"2024-09-17T05:27:54.769713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749303+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:48.749303+0000","last_peered":"2024-09-17T05:27:48.749303+0000","last_clean":"2024-09-17T05:27:48.749303+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:48.749303+0000","last_undegraded":"2024-09-17T05:27:48.749303+0000","last_fullsized":"2024-09-17T05:27:48.749303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"25'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686482+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686482+0000","last_peered":"2024-09-17T05:27:31.686482+0000","last_clean":"2024-09-17T05:27:31.686482+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686482+0000","last_undegraded":"2024-09-17T05:27:31.686482+0000","last_fullsized":"2024-09-17T05:27:31.686482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747002+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:48.747002+0000","last_peered":"2024-09-17T05:27:48.747002+0000","last_clean":"2024-09-17T05:27:48.747002+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:48.747002+0000","last_undegraded":"2024-09-17T05:27:48.747002+0000","last_fullsized":"2024-09-17T05:27:48.747002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.748010+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:48.748010+0000","last_peered":"2024-09-17T05:27:48.748010+0000","last_clean":"2024-09-17T05:27:48.748010+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:48.748010+0000","last_undegraded":"2024-09-17T05:27:48.748010+0000","last_fullsized":"2024-09-17T05:27:48.748010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750067+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:48.750067+0000","last_peered":"2024-09-17T05:27:48.750067+0000","last_clean":"2024-09-17T05:27:48.750067+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:48.750067+0000","last_undegraded":"2024-09-17T05:27:48.750067+0000","last_fullsized":"2024-09-17T05:27:48.750067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770342+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:54.770342+0000","last_peered":"2024-09-17T05:27:54.770342+0000","last_clean":"2024-09-17T05:27:54.770342+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:54.770342+0000","last_undegraded":"2024-09-17T05:27:54.770342+0000","last_fullsized":"2024-09-17T05:27:54.770342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490238+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490238+0000","last_peered":"2024-09-17T05:27:32.490238+0000","last_clean":"2024-09-17T05:27:32.490238+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490238+0000","last_undegraded":"2024-09-17T05:27:32.490238+0000","last_fullsized":"2024-09-17T05:27:32.490238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490659+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490659+0000","last_peered":"2024-09-17T05:27:32.490659+0000","last_clean":"2024-09-17T05:27:32.490659+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490659+0000","last_undegraded":"2024-09-17T05:27:32.490659+0000","last_fullsized":"2024-09-17T05:27:32.490659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771174+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:54.771174+0000","last_peered":"2024-09-17T05:27:54.771174+0000","last_clean":"2024-09-17T05:27:54.771174+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:54.771174+0000","last_undegraded":"2024-09-17T05:27:54.771174+0000","last_fullsized":"2024-09-17T05:27:54.771174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.769978+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:54.769978+0000","last_peered":"2024-09-17T05:27:54.769978+0000","last_clean":"2024-09-17T05:27:54.769978+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:54.769978+0000","last_undegraded":"2024-09-17T05:27:54.769978+0000","last_fullsized":"2024-09-17T05:27:54.769978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750417+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:48.750417+0000","last_peered":"2024-09-17T05:27:48.750417+0000","last_clean":"2024-09-17T05:27:48.750417+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:48.750417+0000","last_undegraded":"2024-09-17T05:27:48.750417+0000","last_fullsized":"2024-09-17T05:27:48.750417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"25'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770792+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:54.770792+0000","last_peered":"2024-09-17T05:27:54.770792+0000","last_clean":"2024-09-17T05:27:54.770792+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:54.770792+0000","last_undegraded":"2024-09-17T05:27:54.770792+0000","last_fullsized":"2024-09-17T05:27:54.770792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489689+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489689+0000","last_peered":"2024-09-17T05:27:32.489689+0000","last_clean":"2024-09-17T05:27:32.489689+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489689+0000","last_undegraded":"2024-09-17T05:27:32.489689+0000","last_fullsized":"2024-09-17T05:27:32.489689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"25'15","reported_seq":38,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771139+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:54.771139+0000","last_peered":"2024-09-17T05:27:54.771139+0000","last_clean":"2024-09-17T05:27:54.771139+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:54.771139+0000","last_undegraded":"2024-09-17T05:27:54.771139+0000","last_fullsized":"2024-09-17T05:27:54.771139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"25'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770264+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:54.770264+0000","last_peered":"2024-09-17T05:27:54.770264+0000","last_clean":"2024-09-17T05:27:54.770264+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:54.770264+0000","last_undegraded":"2024-09-17T05:27:54.770264+0000","last_fullsized":"2024-09-17T05:27:54.770264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749241+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:48.749241+0000","last_peered":"2024-09-17T05:27:48.749241+0000","last_clean":"2024-09-17T05:27:48.749241+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:48.749241+0000","last_undegraded":"2024-09-17T05:27:48.749241+0000","last_fullsized":"2024-09-17T05:27:48.749241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"25'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770268+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:54.770268+0000","last_peered":"2024-09-17T05:27:54.770268+0000","last_clean":"2024-09-17T05:27:54.770268+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:54.770268+0000","last_undegraded":"2024-09-17T05:27:54.770268+0000","last_fullsized":"2024-09-17T05:27:54.770268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'11","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747548+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:48.747548+0000","last_peered":"2024-09-17T05:27:48.747548+0000","last_clean":"2024-09-17T05:27:48.747548+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:48.747548+0000","last_undegraded":"2024-09-17T05:27:48.747548+0000","last_fullsized":"2024-09-17T05:27:48.747548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488889+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488889+0000","last_peered":"2024-09-17T05:27:32.488889+0000","last_clean":"2024-09-17T05:27:32.488889+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488889+0000","last_undegraded":"2024-09-17T05:27:32.488889+0000","last_fullsized":"2024-09-17T05:27:32.488889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746302+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:48.746302+0000","last_peered":"2024-09-17T05:27:48.746302+0000","last_clean":"2024-09-17T05:27:48.746302+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:48.746302+0000","last_undegraded":"2024-09-17T05:27:48.746302+0000","last_fullsized":"2024-09-17T05:27:48.746302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747034+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:48.747034+0000","last_peered":"2024-09-17T05:27:48.747034+0000","last_clean":"2024-09-17T05:27:48.747034+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:48.747034+0000","last_undegraded":"2024-09-17T05:27:48.747034+0000","last_fullsized":"2024-09-17T05:27:48.747034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488921+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488921+0000","last_peered":"2024-09-17T05:27:32.488921+0000","last_clean":"2024-09-17T05:27:32.488921+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488921+0000","last_undegraded":"2024-09-17T05:27:32.488921+0000","last_fullsized":"2024-09-17T05:27:32.488921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771114+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:54.771114+0000","last_peered":"2024-09-17T05:27:54.771114+0000","last_clean":"2024-09-17T05:27:54.771114+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:54.771114+0000","last_undegraded":"2024-09-17T05:27:54.771114+0000","last_fullsized":"2024-09-17T05:27:54.771114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.767532+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:54.767532+0000","last_peered":"2024-09-17T05:27:54.767532+0000","last_clean":"2024-09-17T05:27:54.767532+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:54.767532+0000","last_undegraded":"2024-09-17T05:27:54.767532+0000","last_fullsized":"2024-09-17T05:27:54.767532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"25'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.745637+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:48.745637+0000","last_peered":"2024-09-17T05:27:48.745637+0000","last_clean":"2024-09-17T05:27:48.745637+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:48.745637+0000","last_undegraded":"2024-09-17T05:27:48.745637+0000","last_fullsized":"2024-09-17T05:27:48.745637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":878,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7708,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":15416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":692591,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":103379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":102971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":303104,"data_stored":107473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":693235,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:28:05.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:28:05.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 -- 172.21.15.117:0/624464562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc070890 msgr2=0x7f39bc070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 --2- 172.21.15.117:0/624464562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc070890 0x7f39bc070ca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0c4460 tx=0x7f39b402aa30 comp rx=0 tx=0).stop 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 -- 172.21.15.117:0/624464562 shutdown_connections 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 --2- 172.21.15.117:0/624464562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc070890 0x7f39bc070ca0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 --2- 172.21.15.117:0/624464562 >> v2:172.21.15.119:3300/0 conn(0x7f39bc0c8990 0x7f39bc0c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 --2- 172.21.15.117:0/624464562 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f39bc1581e0 0x7f39bc1516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 -- 172.21.15.117:0/624464562 >> 172.21.15.117:0/624464562 conn(0x7f39bc06d190 msgr2=0x7f39bc06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 -- 172.21.15.117:0/624464562 shutdown_connections 2024-09-17T05:28:05.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.642+0000 7f39c212b640 1 -- 172.21.15.117:0/624464562 wait complete. 2024-09-17T05:28:05.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.644+0000 7f39c212b640 1 Processor -- start 2024-09-17T05:28:05.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.644+0000 7f39c212b640 1 -- start start 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39c212b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f39bc070890 0x7f39bc142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39c212b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 0x7f39bc1eef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39c212b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc1581e0 0x7f39bc140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39c212b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc0c54a0 con 0x7f39bc0c8990 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39c212b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc0733c0 con 0x7f39bc1581e0 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39c212b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc0c5620 con 0x7f39bc070890 2024-09-17T05:28:05.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39baffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 0x7f39bc1eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39baffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 0x7f39bc1eef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40456/0 (socket says 172.21.15.117:40456) 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39baffd640 1 -- 172.21.15.117:0/55347171 learned_addr learned my addr 172.21.15.117:0/55347171 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39bbfff640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc1581e0 0x7f39bc140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.645+0000 7f39bb7fe640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f39bc070890 0x7f39bc142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39baffd640 1 -- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f39bc070890 msgr2=0x7f39bc142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39baffd640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f39bc070890 0x7f39bc142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39baffd640 1 -- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc1581e0 msgr2=0x7f39bc140a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39baffd640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc1581e0 0x7f39bc140a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39baffd640 1 -- 172.21.15.117:0/55347171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc1427c0 con 0x7f39bc0c8990 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39baffd640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 0x7f39bc1eef70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39b000d5e0 tx=0x7f39b000daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0017070 con 0x7f39bc0c8990 2024-09-17T05:28:05.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39b0004310 con 0x7f39bc0c8990 2024-09-17T05:28:05.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39bc146be0 con 0x7f39bc0c8990 2024-09-17T05:28:05.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0005260 con 0x7f39bc0c8990 2024-09-17T05:28:05.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.646+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39bc1429d0 con 0x7f39bc0c8990 2024-09-17T05:28:05.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.647+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39bc0c3820 con 0x7f39bc0c8990 2024-09-17T05:28:05.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.648+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f39b0005400 con 0x7f39bc0c8990 2024-09-17T05:28:05.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.648+0000 7f39a3fff640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39a8078bc0 0x7f39a807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:05.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.648+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f39b009b220 con 0x7f39bc0c8990 2024-09-17T05:28:05.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.649+0000 7f39bb7fe640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39a8078bc0 0x7f39a807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:05.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.649+0000 7f39bb7fe640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39a8078bc0 0x7f39a807b0d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00bb50 tx=0x7f39ac00b2e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:05.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.653+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f39b0020020 con 0x7f39bc0c8990 2024-09-17T05:28:05.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.826+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39bc0c4dd0 con 0x7f39a8078bc0 2024-09-17T05:28:05.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.832+0000 7f39a3fff640 1 -- 172.21.15.117:0/55347171 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332664 (secure 0 0 0) 0x7f39bc0c4dd0 con 0x7f39a8078bc0 2024-09-17T05:28:05.839 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:05.842 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:28:05.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39a8078bc0 msgr2=0x7f39a807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39a8078bc0 0x7f39a807b0d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00bb50 tx=0x7f39ac00b2e0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 msgr2=0x7f39bc1eef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 0x7f39bc1eef70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f39b000d5e0 tx=0x7f39b000daa0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 shutdown_connections 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39a8078bc0 0x7f39a807b0d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc1581e0 0x7f39bc140a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39bc0c8990 0x7f39bc1eef70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 --2- 172.21.15.117:0/55347171 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f39bc070890 0x7f39bc142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 >> 172.21.15.117:0/55347171 conn(0x7f39bc06d190 msgr2=0x7f39bc0c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 shutdown_connections 2024-09-17T05:28:05.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:05.836+0000 7f39c212b640 1 -- 172.21.15.117:0/55347171 wait complete. 2024-09-17T05:28:05.848 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-17T05:28:05.823238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598076,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":230,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218496,"kb_used_data":2304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756224,"statfs":{"total":773094113280,"available":772870373376,"internally_reserved":0,"allocated":2359296,"data_stored":2012400,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1040,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.466773"},"pg_stats":[{"pgid":"2.1f","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770490+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:54.770490+0000","last_peered":"2024-09-17T05:27:54.770490+0000","last_clean":"2024-09-17T05:27:54.770490+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:54.770490+0000","last_undegraded":"2024-09-17T05:27:54.770490+0000","last_fullsized":"2024-09-17T05:27:54.770490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688890+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688890+0000","last_peered":"2024-09-17T05:27:31.688890+0000","last_clean":"2024-09-17T05:27:31.688890+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688890+0000","last_undegraded":"2024-09-17T05:27:31.688890+0000","last_fullsized":"2024-09-17T05:27:31.688890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.769713+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:54.769713+0000","last_peered":"2024-09-17T05:27:54.769713+0000","last_clean":"2024-09-17T05:27:54.769713+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:54.769713+0000","last_undegraded":"2024-09-17T05:27:54.769713+0000","last_fullsized":"2024-09-17T05:27:54.769713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749303+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:48.749303+0000","last_peered":"2024-09-17T05:27:48.749303+0000","last_clean":"2024-09-17T05:27:48.749303+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:48.749303+0000","last_undegraded":"2024-09-17T05:27:48.749303+0000","last_fullsized":"2024-09-17T05:27:48.749303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"25'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686482+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686482+0000","last_peered":"2024-09-17T05:27:31.686482+0000","last_clean":"2024-09-17T05:27:31.686482+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686482+0000","last_undegraded":"2024-09-17T05:27:31.686482+0000","last_fullsized":"2024-09-17T05:27:31.686482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747002+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:48.747002+0000","last_peered":"2024-09-17T05:27:48.747002+0000","last_clean":"2024-09-17T05:27:48.747002+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:48.747002+0000","last_undegraded":"2024-09-17T05:27:48.747002+0000","last_fullsized":"2024-09-17T05:27:48.747002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.748010+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:48.748010+0000","last_peered":"2024-09-17T05:27:48.748010+0000","last_clean":"2024-09-17T05:27:48.748010+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:48.748010+0000","last_undegraded":"2024-09-17T05:27:48.748010+0000","last_fullsized":"2024-09-17T05:27:48.748010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750067+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:48.750067+0000","last_peered":"2024-09-17T05:27:48.750067+0000","last_clean":"2024-09-17T05:27:48.750067+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:48.750067+0000","last_undegraded":"2024-09-17T05:27:48.750067+0000","last_fullsized":"2024-09-17T05:27:48.750067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770342+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:54.770342+0000","last_peered":"2024-09-17T05:27:54.770342+0000","last_clean":"2024-09-17T05:27:54.770342+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:54.770342+0000","last_undegraded":"2024-09-17T05:27:54.770342+0000","last_fullsized":"2024-09-17T05:27:54.770342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490238+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490238+0000","last_peered":"2024-09-17T05:27:32.490238+0000","last_clean":"2024-09-17T05:27:32.490238+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490238+0000","last_undegraded":"2024-09-17T05:27:32.490238+0000","last_fullsized":"2024-09-17T05:27:32.490238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490659+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490659+0000","last_peered":"2024-09-17T05:27:32.490659+0000","last_clean":"2024-09-17T05:27:32.490659+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490659+0000","last_undegraded":"2024-09-17T05:27:32.490659+0000","last_fullsized":"2024-09-17T05:27:32.490659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771174+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:54.771174+0000","last_peered":"2024-09-17T05:27:54.771174+0000","last_clean":"2024-09-17T05:27:54.771174+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:54.771174+0000","last_undegraded":"2024-09-17T05:27:54.771174+0000","last_fullsized":"2024-09-17T05:27:54.771174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.769978+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:54.769978+0000","last_peered":"2024-09-17T05:27:54.769978+0000","last_clean":"2024-09-17T05:27:54.769978+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:54.769978+0000","last_undegraded":"2024-09-17T05:27:54.769978+0000","last_fullsized":"2024-09-17T05:27:54.769978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750417+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:48.750417+0000","last_peered":"2024-09-17T05:27:48.750417+0000","last_clean":"2024-09-17T05:27:48.750417+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:48.750417+0000","last_undegraded":"2024-09-17T05:27:48.750417+0000","last_fullsized":"2024-09-17T05:27:48.750417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"25'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770792+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:54.770792+0000","last_peered":"2024-09-17T05:27:54.770792+0000","last_clean":"2024-09-17T05:27:54.770792+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:54.770792+0000","last_undegraded":"2024-09-17T05:27:54.770792+0000","last_fullsized":"2024-09-17T05:27:54.770792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489689+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489689+0000","last_peered":"2024-09-17T05:27:32.489689+0000","last_clean":"2024-09-17T05:27:32.489689+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489689+0000","last_undegraded":"2024-09-17T05:27:32.489689+0000","last_fullsized":"2024-09-17T05:27:32.489689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"25'15","reported_seq":38,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771139+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:54.771139+0000","last_peered":"2024-09-17T05:27:54.771139+0000","last_clean":"2024-09-17T05:27:54.771139+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:54.771139+0000","last_undegraded":"2024-09-17T05:27:54.771139+0000","last_fullsized":"2024-09-17T05:27:54.771139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"25'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770264+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:54.770264+0000","last_peered":"2024-09-17T05:27:54.770264+0000","last_clean":"2024-09-17T05:27:54.770264+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:54.770264+0000","last_undegraded":"2024-09-17T05:27:54.770264+0000","last_fullsized":"2024-09-17T05:27:54.770264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749241+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:48.749241+0000","last_peered":"2024-09-17T05:27:48.749241+0000","last_clean":"2024-09-17T05:27:48.749241+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:48.749241+0000","last_undegraded":"2024-09-17T05:27:48.749241+0000","last_fullsized":"2024-09-17T05:27:48.749241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"25'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770268+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:54.770268+0000","last_peered":"2024-09-17T05:27:54.770268+0000","last_clean":"2024-09-17T05:27:54.770268+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:54.770268+0000","last_undegraded":"2024-09-17T05:27:54.770268+0000","last_fullsized":"2024-09-17T05:27:54.770268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'11","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747548+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:48.747548+0000","last_peered":"2024-09-17T05:27:48.747548+0000","last_clean":"2024-09-17T05:27:48.747548+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:48.747548+0000","last_undegraded":"2024-09-17T05:27:48.747548+0000","last_fullsized":"2024-09-17T05:27:48.747548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488889+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488889+0000","last_peered":"2024-09-17T05:27:32.488889+0000","last_clean":"2024-09-17T05:27:32.488889+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488889+0000","last_undegraded":"2024-09-17T05:27:32.488889+0000","last_fullsized":"2024-09-17T05:27:32.488889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746302+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:48.746302+0000","last_peered":"2024-09-17T05:27:48.746302+0000","last_clean":"2024-09-17T05:27:48.746302+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:48.746302+0000","last_undegraded":"2024-09-17T05:27:48.746302+0000","last_fullsized":"2024-09-17T05:27:48.746302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747034+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:48.747034+0000","last_peered":"2024-09-17T05:27:48.747034+0000","last_clean":"2024-09-17T05:27:48.747034+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:48.747034+0000","last_undegraded":"2024-09-17T05:27:48.747034+0000","last_fullsized":"2024-09-17T05:27:48.747034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488921+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488921+0000","last_peered":"2024-09-17T05:27:32.488921+0000","last_clean":"2024-09-17T05:27:32.488921+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488921+0000","last_undegraded":"2024-09-17T05:27:32.488921+0000","last_fullsized":"2024-09-17T05:27:32.488921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771114+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:54.771114+0000","last_peered":"2024-09-17T05:27:54.771114+0000","last_clean":"2024-09-17T05:27:54.771114+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:54.771114+0000","last_undegraded":"2024-09-17T05:27:54.771114+0000","last_fullsized":"2024-09-17T05:27:54.771114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.767532+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:54.767532+0000","last_peered":"2024-09-17T05:27:54.767532+0000","last_clean":"2024-09-17T05:27:54.767532+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:54.767532+0000","last_undegraded":"2024-09-17T05:27:54.767532+0000","last_fullsized":"2024-09-17T05:27:54.767532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"25'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.745637+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:48.745637+0000","last_peered":"2024-09-17T05:27:48.745637+0000","last_clean":"2024-09-17T05:27:48.745637+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:48.745637+0000","last_undegraded":"2024-09-17T05:27:48.745637+0000","last_fullsized":"2024-09-17T05:27:48.745637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":878,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7708,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":15416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":692591,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":103379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":102971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":303104,"data_stored":107473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":693235,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:28:05.854 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-09-17T05:28:05.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:28:06.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813918e640 1 -- 172.21.15.117:0/3339910308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f811c003a30 msgr2=0x7f811c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813918e640 1 --2- 172.21.15.117:0/3339910308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f811c003a30 0x7f811c003ea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f812400c260 tx=0x7f811c005830 comp rx=0 tx=0).stop 2024-09-17T05:28:06.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813359e640 1 -- 172.21.15.117:0/3339910308 reap_dead start 2024-09-17T05:28:06.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813918e640 1 -- 172.21.15.117:0/3339910308 shutdown_connections 2024-09-17T05:28:06.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813918e640 1 -- 172.21.15.117:0/3339910308 >> 172.21.15.117:0/3339910308 conn(0x7f813406cda0 msgr2=0x7f813406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:06.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813918e640 1 -- 172.21.15.117:0/3339910308 shutdown_connections 2024-09-17T05:28:06.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.000+0000 7f813918e640 1 -- 172.21.15.117:0/3339910308 wait complete. 2024-09-17T05:28:06.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.002+0000 7f813918e640 1 Processor -- start 2024-09-17T05:28:06.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.002+0000 7f813918e640 1 -- start start 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.003+0000 7f813918e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f811c003a30 0x7f8134158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.003+0000 7f813918e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 0x7f81341f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.003+0000 7f813918e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81340c5500 0x7f813414b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.003+0000 7f813918e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f81340c3ab0 con 0x7f8134157ec0 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.003+0000 7f813918e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f813406e0c0 con 0x7f81340c5500 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.003+0000 7f813918e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f8134075e00 con 0x7f811c003a30 2024-09-17T05:28:06.006 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 0x7f81341f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 0x7f81341f1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40476/0 (socket says 172.21.15.117:40476) 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 -- 172.21.15.117:0/13053832 learned_addr learned my addr 172.21.15.117:0/13053832 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813359e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81340c5500 0x7f813414b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f8132d9d640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f811c003a30 0x7f8134158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 -- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f811c003a30 msgr2=0x7f8134158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f811c003a30 0x7f8134158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 -- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81340c5500 msgr2=0x7f813414b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81340c5500 0x7f813414b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 -- 172.21.15.117:0/13053832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8134157010 con 0x7f8134157ec0 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813259c640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 0x7f81341f1a30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f81280118b0 tx=0x7f8128011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812801b070 con 0x7f8134157ec0 2024-09-17T05:28:06.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f812800f940 con 0x7f8134157ec0 2024-09-17T05:28:06.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81341f1f70 con 0x7f8134157ec0 2024-09-17T05:28:06.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812800fd90 con 0x7f8134157ec0 2024-09-17T05:28:06.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.004+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81341557f0 con 0x7f8134157ec0 2024-09-17T05:28:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.005+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8134158c70 con 0x7f8134157ec0 2024-09-17T05:28:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.006+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8128016070 con 0x7f8134157ec0 2024-09-17T05:28:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.007+0000 7f811b7fe640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8120078b90 0x7f812007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.007+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f812809dd80 con 0x7f8134157ec0 2024-09-17T05:28:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.007+0000 7f8132d9d640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8120078b90 0x7f812007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.007+0000 7f8132d9d640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8120078b90 0x7f812007b0a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8124033000 tx=0x7f8124038040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:06.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.011+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8134157010 con 0x7f8134157ec0 2024-09-17T05:28:06.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.183+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81340c2d10 con 0x7f8120078b90 2024-09-17T05:28:06.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.189+0000 7f811b7fe640 1 -- 172.21.15.117:0/13053832 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332664 (secure 0 0 0) 0x7f81340c2d10 con 0x7f8120078b90 2024-09-17T05:28:06.196 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:06.198 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:28:06.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8120078b90 msgr2=0x7f812007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8120078b90 0x7f812007b0a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8124033000 tx=0x7f8124038040 comp rx=0 tx=0).stop 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 msgr2=0x7f81341f1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 0x7f81341f1a30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f81280118b0 tx=0x7f8128011d70 comp rx=0 tx=0).stop 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 shutdown_connections 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8120078b90 0x7f812007b0a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81340c5500 0x7f813414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8134157ec0 0x7f81341f1a30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 --2- 172.21.15.117:0/13053832 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f811c003a30 0x7f8134158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.192+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 >> 172.21.15.117:0/13053832 conn(0x7f813406cda0 msgr2=0x7f81340cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.193+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 shutdown_connections 2024-09-17T05:28:06.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.193+0000 7f813918e640 1 -- 172.21.15.117:0/13053832 wait complete. 2024-09-17T05:28:06.205 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-17T05:28:05.823238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598076,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":230,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218496,"kb_used_data":2304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756224,"statfs":{"total":773094113280,"available":772870373376,"internally_reserved":0,"allocated":2359296,"data_stored":2012400,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1040,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.466773"},"pg_stats":[{"pgid":"2.1f","version":"25'9","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770490+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:27:54.770490+0000","last_peered":"2024-09-17T05:27:54.770490+0000","last_clean":"2024-09-17T05:27:54.770490+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:27:54.770490+0000","last_undegraded":"2024-09-17T05:27:54.770490+0000","last_fullsized":"2024-09-17T05:27:54.770490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688686+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:27:31.688686+0000","last_peered":"2024-09-17T05:27:31.688686+0000","last_clean":"2024-09-17T05:27:31.688686+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:27:31.688686+0000","last_undegraded":"2024-09-17T05:27:31.688686+0000","last_fullsized":"2024-09-17T05:27:31.688686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488326+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:27:32.488326+0000","last_peered":"2024-09-17T05:27:32.488326+0000","last_clean":"2024-09-17T05:27:32.488326+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:27:32.488326+0000","last_undegraded":"2024-09-17T05:27:32.488326+0000","last_fullsized":"2024-09-17T05:27:32.488326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688890+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:27:31.688890+0000","last_peered":"2024-09-17T05:27:31.688890+0000","last_clean":"2024-09-17T05:27:31.688890+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:27:31.688890+0000","last_undegraded":"2024-09-17T05:27:31.688890+0000","last_fullsized":"2024-09-17T05:27:31.688890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688214+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:27:31.688214+0000","last_peered":"2024-09-17T05:27:31.688214+0000","last_clean":"2024-09-17T05:27:31.688214+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:27:31.688214+0000","last_undegraded":"2024-09-17T05:27:31.688214+0000","last_fullsized":"2024-09-17T05:27:31.688214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487938+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:27:32.487938+0000","last_peered":"2024-09-17T05:27:32.487938+0000","last_clean":"2024-09-17T05:27:32.487938+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:27:32.487938+0000","last_undegraded":"2024-09-17T05:27:32.487938+0000","last_fullsized":"2024-09-17T05:27:32.487938+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.769713+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:27:54.769713+0000","last_peered":"2024-09-17T05:27:54.769713+0000","last_clean":"2024-09-17T05:27:54.769713+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:27:54.769713+0000","last_undegraded":"2024-09-17T05:27:54.769713+0000","last_fullsized":"2024-09-17T05:27:54.769713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763790+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:27:31.763790+0000","last_peered":"2024-09-17T05:27:31.763790+0000","last_clean":"2024-09-17T05:27:31.763790+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:27:31.763790+0000","last_undegraded":"2024-09-17T05:27:31.763790+0000","last_fullsized":"2024-09-17T05:27:31.763790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488635+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:27:32.488635+0000","last_peered":"2024-09-17T05:27:32.488635+0000","last_clean":"2024-09-17T05:27:32.488635+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:27:32.488635+0000","last_undegraded":"2024-09-17T05:27:32.488635+0000","last_fullsized":"2024-09-17T05:27:32.488635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"24'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490120+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:27:32.490120+0000","last_peered":"2024-09-17T05:27:32.490120+0000","last_clean":"2024-09-17T05:27:32.490120+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:27:32.490120+0000","last_undegraded":"2024-09-17T05:27:32.490120+0000","last_fullsized":"2024-09-17T05:27:32.490120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489651+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:27:32.489651+0000","last_peered":"2024-09-17T05:27:32.489651+0000","last_clean":"2024-09-17T05:27:32.489651+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:27:32.489651+0000","last_undegraded":"2024-09-17T05:27:32.489651+0000","last_fullsized":"2024-09-17T05:27:32.489651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.487936+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:27:32.487936+0000","last_peered":"2024-09-17T05:27:32.487936+0000","last_clean":"2024-09-17T05:27:32.487936+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:27:32.487936+0000","last_undegraded":"2024-09-17T05:27:32.487936+0000","last_fullsized":"2024-09-17T05:27:32.487936+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490353+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:27:32.490353+0000","last_peered":"2024-09-17T05:27:32.490353+0000","last_clean":"2024-09-17T05:27:32.490353+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:27:32.490353+0000","last_undegraded":"2024-09-17T05:27:32.490353+0000","last_fullsized":"2024-09-17T05:27:32.490353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749303+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:27:48.749303+0000","last_peered":"2024-09-17T05:27:48.749303+0000","last_clean":"2024-09-17T05:27:48.749303+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:27:48.749303+0000","last_undegraded":"2024-09-17T05:27:48.749303+0000","last_fullsized":"2024-09-17T05:27:48.749303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489162+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:27:32.489162+0000","last_peered":"2024-09-17T05:27:32.489162+0000","last_clean":"2024-09-17T05:27:32.489162+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:27:32.489162+0000","last_undegraded":"2024-09-17T05:27:32.489162+0000","last_fullsized":"2024-09-17T05:27:32.489162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488608+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:27:32.488608+0000","last_peered":"2024-09-17T05:27:32.488608+0000","last_clean":"2024-09-17T05:27:32.488608+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:27:32.488608+0000","last_undegraded":"2024-09-17T05:27:32.488608+0000","last_fullsized":"2024-09-17T05:27:32.488608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"25'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686482+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:27:31.686482+0000","last_peered":"2024-09-17T05:27:31.686482+0000","last_clean":"2024-09-17T05:27:31.686482+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:27:31.686482+0000","last_undegraded":"2024-09-17T05:27:31.686482+0000","last_fullsized":"2024-09-17T05:27:31.686482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489437+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:27:32.489437+0000","last_peered":"2024-09-17T05:27:32.489437+0000","last_clean":"2024-09-17T05:27:32.489437+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:27:32.489437+0000","last_undegraded":"2024-09-17T05:27:32.489437+0000","last_fullsized":"2024-09-17T05:27:32.489437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687740+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:27:31.687740+0000","last_peered":"2024-09-17T05:27:31.687740+0000","last_clean":"2024-09-17T05:27:31.687740+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:27:31.687740+0000","last_undegraded":"2024-09-17T05:27:31.687740+0000","last_fullsized":"2024-09-17T05:27:31.687740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747002+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:27:48.747002+0000","last_peered":"2024-09-17T05:27:48.747002+0000","last_clean":"2024-09-17T05:27:48.747002+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:27:48.747002+0000","last_undegraded":"2024-09-17T05:27:48.747002+0000","last_fullsized":"2024-09-17T05:27:48.747002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488720+0000","last_change":"2024-09-17T05:27:30.691769+0000","last_active":"2024-09-17T05:27:32.488720+0000","last_peered":"2024-09-17T05:27:32.488720+0000","last_clean":"2024-09-17T05:27:32.488720+0000","last_became_active":"2024-09-17T05:27:30.690968+0000","last_became_peered":"2024-09-17T05:27:30.690968+0000","last_unstale":"2024-09-17T05:27:32.488720+0000","last_undegraded":"2024-09-17T05:27:32.488720+0000","last_fullsized":"2024-09-17T05:27:32.488720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:02.103985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2,3],"acting":[7,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488787+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:27:32.488787+0000","last_peered":"2024-09-17T05:27:32.488787+0000","last_clean":"2024-09-17T05:27:32.488787+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:27:32.488787+0000","last_undegraded":"2024-09-17T05:27:32.488787+0000","last_fullsized":"2024-09-17T05:27:32.488787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.748010+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:27:48.748010+0000","last_peered":"2024-09-17T05:27:48.748010+0000","last_clean":"2024-09-17T05:27:48.748010+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:27:48.748010+0000","last_undegraded":"2024-09-17T05:27:48.748010+0000","last_fullsized":"2024-09-17T05:27:48.748010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488963+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:27:32.488963+0000","last_peered":"2024-09-17T05:27:32.488963+0000","last_clean":"2024-09-17T05:27:32.488963+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:27:32.488963+0000","last_undegraded":"2024-09-17T05:27:32.488963+0000","last_fullsized":"2024-09-17T05:27:32.488963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686422+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:27:31.686422+0000","last_peered":"2024-09-17T05:27:31.686422+0000","last_clean":"2024-09-17T05:27:31.686422+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:27:31.686422+0000","last_undegraded":"2024-09-17T05:27:31.686422+0000","last_fullsized":"2024-09-17T05:27:31.686422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"25'4","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750067+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:27:48.750067+0000","last_peered":"2024-09-17T05:27:48.750067+0000","last_clean":"2024-09-17T05:27:48.750067+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:27:48.750067+0000","last_undegraded":"2024-09-17T05:27:48.750067+0000","last_fullsized":"2024-09-17T05:27:48.750067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488980+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:27:32.488980+0000","last_peered":"2024-09-17T05:27:32.488980+0000","last_clean":"2024-09-17T05:27:32.488980+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:27:32.488980+0000","last_undegraded":"2024-09-17T05:27:32.488980+0000","last_fullsized":"2024-09-17T05:27:32.488980+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688128+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:27:31.688128+0000","last_peered":"2024-09-17T05:27:31.688128+0000","last_clean":"2024-09-17T05:27:31.688128+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:27:31.688128+0000","last_undegraded":"2024-09-17T05:27:31.688128+0000","last_fullsized":"2024-09-17T05:27:31.688128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770342+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:27:54.770342+0000","last_peered":"2024-09-17T05:27:54.770342+0000","last_clean":"2024-09-17T05:27:54.770342+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:27:54.770342+0000","last_undegraded":"2024-09-17T05:27:54.770342+0000","last_fullsized":"2024-09-17T05:27:54.770342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687655+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:27:31.687655+0000","last_peered":"2024-09-17T05:27:31.687655+0000","last_clean":"2024-09-17T05:27:31.687655+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:27:31.687655+0000","last_undegraded":"2024-09-17T05:27:31.687655+0000","last_fullsized":"2024-09-17T05:27:31.687655+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687571+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:27:31.687571+0000","last_peered":"2024-09-17T05:27:31.687571+0000","last_clean":"2024-09-17T05:27:31.687571+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:27:31.687571+0000","last_undegraded":"2024-09-17T05:27:31.687571+0000","last_fullsized":"2024-09-17T05:27:31.687571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490238+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:27:32.490238+0000","last_peered":"2024-09-17T05:27:32.490238+0000","last_clean":"2024-09-17T05:27:32.490238+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:27:32.490238+0000","last_undegraded":"2024-09-17T05:27:32.490238+0000","last_fullsized":"2024-09-17T05:27:32.490238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686854+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:27:31.686854+0000","last_peered":"2024-09-17T05:27:31.686854+0000","last_clean":"2024-09-17T05:27:31.686854+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:27:31.686854+0000","last_undegraded":"2024-09-17T05:27:31.686854+0000","last_fullsized":"2024-09-17T05:27:31.686854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488069+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:27:32.488069+0000","last_peered":"2024-09-17T05:27:32.488069+0000","last_clean":"2024-09-17T05:27:32.488069+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:27:32.488069+0000","last_undegraded":"2024-09-17T05:27:32.488069+0000","last_fullsized":"2024-09-17T05:27:32.488069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687303+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:27:31.687303+0000","last_peered":"2024-09-17T05:27:31.687303+0000","last_clean":"2024-09-17T05:27:31.687303+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:27:31.687303+0000","last_undegraded":"2024-09-17T05:27:31.687303+0000","last_fullsized":"2024-09-17T05:27:31.687303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686052+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:27:31.686052+0000","last_peered":"2024-09-17T05:27:31.686052+0000","last_clean":"2024-09-17T05:27:31.686052+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:27:31.686052+0000","last_undegraded":"2024-09-17T05:27:31.686052+0000","last_fullsized":"2024-09-17T05:27:31.686052+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488945+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:27:32.488945+0000","last_peered":"2024-09-17T05:27:32.488945+0000","last_clean":"2024-09-17T05:27:32.488945+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:27:32.488945+0000","last_undegraded":"2024-09-17T05:27:32.488945+0000","last_fullsized":"2024-09-17T05:27:32.488945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490659+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:27:32.490659+0000","last_peered":"2024-09-17T05:27:32.490659+0000","last_clean":"2024-09-17T05:27:32.490659+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:27:32.490659+0000","last_undegraded":"2024-09-17T05:27:32.490659+0000","last_fullsized":"2024-09-17T05:27:32.490659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687316+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:27:31.687316+0000","last_peered":"2024-09-17T05:27:31.687316+0000","last_clean":"2024-09-17T05:27:31.687316+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:27:31.687316+0000","last_undegraded":"2024-09-17T05:27:31.687316+0000","last_fullsized":"2024-09-17T05:27:31.687316+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686356+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:27:31.686356+0000","last_peered":"2024-09-17T05:27:31.686356+0000","last_clean":"2024-09-17T05:27:31.686356+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:27:31.686356+0000","last_undegraded":"2024-09-17T05:27:31.686356+0000","last_fullsized":"2024-09-17T05:27:31.686356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771174+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:27:54.771174+0000","last_peered":"2024-09-17T05:27:54.771174+0000","last_clean":"2024-09-17T05:27:54.771174+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:27:54.771174+0000","last_undegraded":"2024-09-17T05:27:54.771174+0000","last_fullsized":"2024-09-17T05:27:54.771174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686449+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:27:31.686449+0000","last_peered":"2024-09-17T05:27:31.686449+0000","last_clean":"2024-09-17T05:27:31.686449+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:27:31.686449+0000","last_undegraded":"2024-09-17T05:27:31.686449+0000","last_fullsized":"2024-09-17T05:27:31.686449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685963+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:27:31.685963+0000","last_peered":"2024-09-17T05:27:31.685963+0000","last_clean":"2024-09-17T05:27:31.685963+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:27:31.685963+0000","last_undegraded":"2024-09-17T05:27:31.685963+0000","last_fullsized":"2024-09-17T05:27:31.685963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.769978+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:27:54.769978+0000","last_peered":"2024-09-17T05:27:54.769978+0000","last_clean":"2024-09-17T05:27:54.769978+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:27:54.769978+0000","last_undegraded":"2024-09-17T05:27:54.769978+0000","last_fullsized":"2024-09-17T05:27:54.769978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688472+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:27:31.688472+0000","last_peered":"2024-09-17T05:27:31.688472+0000","last_clean":"2024-09-17T05:27:31.688472+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:27:31.688472+0000","last_undegraded":"2024-09-17T05:27:31.688472+0000","last_fullsized":"2024-09-17T05:27:31.688472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688516+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:27:31.688516+0000","last_peered":"2024-09-17T05:27:31.688516+0000","last_clean":"2024-09-17T05:27:31.688516+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:27:31.688516+0000","last_undegraded":"2024-09-17T05:27:31.688516+0000","last_fullsized":"2024-09-17T05:27:31.688516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.750417+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:27:48.750417+0000","last_peered":"2024-09-17T05:27:48.750417+0000","last_clean":"2024-09-17T05:27:48.750417+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:27:48.750417+0000","last_undegraded":"2024-09-17T05:27:48.750417+0000","last_fullsized":"2024-09-17T05:27:48.750417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":119,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763574+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:27:31.763574+0000","last_peered":"2024-09-17T05:27:31.763574+0000","last_clean":"2024-09-17T05:27:31.763574+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:27:31.763574+0000","last_undegraded":"2024-09-17T05:27:31.763574+0000","last_fullsized":"2024-09-17T05:27:31.763574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489923+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:27:32.489923+0000","last_peered":"2024-09-17T05:27:32.489923+0000","last_clean":"2024-09-17T05:27:32.489923+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:27:32.489923+0000","last_undegraded":"2024-09-17T05:27:32.489923+0000","last_fullsized":"2024-09-17T05:27:32.489923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489754+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:27:32.489754+0000","last_peered":"2024-09-17T05:27:32.489754+0000","last_clean":"2024-09-17T05:27:32.489754+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:27:32.489754+0000","last_undegraded":"2024-09-17T05:27:32.489754+0000","last_fullsized":"2024-09-17T05:27:32.489754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"25'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770792+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:27:54.770792+0000","last_peered":"2024-09-17T05:27:54.770792+0000","last_clean":"2024-09-17T05:27:54.770792+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:27:54.770792+0000","last_undegraded":"2024-09-17T05:27:54.770792+0000","last_fullsized":"2024-09-17T05:27:54.770792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488294+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:27:32.488294+0000","last_peered":"2024-09-17T05:27:32.488294+0000","last_clean":"2024-09-17T05:27:32.488294+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:27:32.488294+0000","last_undegraded":"2024-09-17T05:27:32.488294+0000","last_fullsized":"2024-09-17T05:27:32.488294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490035+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:27:32.490035+0000","last_peered":"2024-09-17T05:27:32.490035+0000","last_clean":"2024-09-17T05:27:32.490035+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:27:32.490035+0000","last_undegraded":"2024-09-17T05:27:32.490035+0000","last_fullsized":"2024-09-17T05:27:32.490035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"24'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687579+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:27:31.687579+0000","last_peered":"2024-09-17T05:27:31.687579+0000","last_clean":"2024-09-17T05:27:31.687579+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:27:31.687579+0000","last_undegraded":"2024-09-17T05:27:31.687579+0000","last_fullsized":"2024-09-17T05:27:31.687579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686803+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:27:31.686803+0000","last_peered":"2024-09-17T05:27:31.686803+0000","last_clean":"2024-09-17T05:27:31.686803+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:27:31.686803+0000","last_undegraded":"2024-09-17T05:27:31.686803+0000","last_fullsized":"2024-09-17T05:27:31.686803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489412+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:27:32.489412+0000","last_peered":"2024-09-17T05:27:32.489412+0000","last_clean":"2024-09-17T05:27:32.489412+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:27:32.489412+0000","last_undegraded":"2024-09-17T05:27:32.489412+0000","last_fullsized":"2024-09-17T05:27:32.489412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"24'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686539+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:27:31.686539+0000","last_peered":"2024-09-17T05:27:31.686539+0000","last_clean":"2024-09-17T05:27:31.686539+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:27:31.686539+0000","last_undegraded":"2024-09-17T05:27:31.686539+0000","last_fullsized":"2024-09-17T05:27:31.686539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488382+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:27:32.488382+0000","last_peered":"2024-09-17T05:27:32.488382+0000","last_clean":"2024-09-17T05:27:32.488382+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:27:32.488382+0000","last_undegraded":"2024-09-17T05:27:32.488382+0000","last_fullsized":"2024-09-17T05:27:32.488382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489200+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:27:32.489200+0000","last_peered":"2024-09-17T05:27:32.489200+0000","last_clean":"2024-09-17T05:27:32.489200+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:27:32.489200+0000","last_undegraded":"2024-09-17T05:27:32.489200+0000","last_fullsized":"2024-09-17T05:27:32.489200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489689+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:27:32.489689+0000","last_peered":"2024-09-17T05:27:32.489689+0000","last_clean":"2024-09-17T05:27:32.489689+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:27:32.489689+0000","last_undegraded":"2024-09-17T05:27:32.489689+0000","last_fullsized":"2024-09-17T05:27:32.489689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763537+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:27:31.763537+0000","last_peered":"2024-09-17T05:27:31.763537+0000","last_clean":"2024-09-17T05:27:31.763537+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:27:31.763537+0000","last_undegraded":"2024-09-17T05:27:31.763537+0000","last_fullsized":"2024-09-17T05:27:31.763537+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488665+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:27:32.488665+0000","last_peered":"2024-09-17T05:27:32.488665+0000","last_clean":"2024-09-17T05:27:32.488665+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:27:32.488665+0000","last_undegraded":"2024-09-17T05:27:32.488665+0000","last_fullsized":"2024-09-17T05:27:32.488665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"25'15","reported_seq":38,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771139+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:27:54.771139+0000","last_peered":"2024-09-17T05:27:54.771139+0000","last_clean":"2024-09-17T05:27:54.771139+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:27:54.771139+0000","last_undegraded":"2024-09-17T05:27:54.771139+0000","last_fullsized":"2024-09-17T05:27:54.771139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688994+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:27:31.688994+0000","last_peered":"2024-09-17T05:27:31.688994+0000","last_clean":"2024-09-17T05:27:31.688994+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:27:31.688994+0000","last_undegraded":"2024-09-17T05:27:31.688994+0000","last_fullsized":"2024-09-17T05:27:31.688994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763343+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:27:31.763343+0000","last_peered":"2024-09-17T05:27:31.763343+0000","last_clean":"2024-09-17T05:27:31.763343+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:27:31.763343+0000","last_undegraded":"2024-09-17T05:27:31.763343+0000","last_fullsized":"2024-09-17T05:27:31.763343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"25'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770264+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:27:54.770264+0000","last_peered":"2024-09-17T05:27:54.770264+0000","last_clean":"2024-09-17T05:27:54.770264+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:27:54.770264+0000","last_undegraded":"2024-09-17T05:27:54.770264+0000","last_fullsized":"2024-09-17T05:27:54.770264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688449+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:27:31.688449+0000","last_peered":"2024-09-17T05:27:31.688449+0000","last_clean":"2024-09-17T05:27:31.688449+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:27:31.688449+0000","last_undegraded":"2024-09-17T05:27:31.688449+0000","last_fullsized":"2024-09-17T05:27:31.688449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489666+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:27:32.489666+0000","last_peered":"2024-09-17T05:27:32.489666+0000","last_clean":"2024-09-17T05:27:32.489666+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:27:32.489666+0000","last_undegraded":"2024-09-17T05:27:32.489666+0000","last_fullsized":"2024-09-17T05:27:32.489666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.749241+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:27:48.749241+0000","last_peered":"2024-09-17T05:27:48.749241+0000","last_clean":"2024-09-17T05:27:48.749241+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:27:48.749241+0000","last_undegraded":"2024-09-17T05:27:48.749241+0000","last_fullsized":"2024-09-17T05:27:48.749241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687064+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:27:31.687064+0000","last_peered":"2024-09-17T05:27:31.687064+0000","last_clean":"2024-09-17T05:27:31.687064+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:27:31.687064+0000","last_undegraded":"2024-09-17T05:27:31.687064+0000","last_fullsized":"2024-09-17T05:27:31.687064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686255+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:27:31.686255+0000","last_peered":"2024-09-17T05:27:31.686255+0000","last_clean":"2024-09-17T05:27:31.686255+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:27:31.686255+0000","last_undegraded":"2024-09-17T05:27:31.686255+0000","last_fullsized":"2024-09-17T05:27:31.686255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"25'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.770268+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:27:54.770268+0000","last_peered":"2024-09-17T05:27:54.770268+0000","last_clean":"2024-09-17T05:27:54.770268+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:27:54.770268+0000","last_undegraded":"2024-09-17T05:27:54.770268+0000","last_fullsized":"2024-09-17T05:27:54.770268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489475+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:27:32.489475+0000","last_peered":"2024-09-17T05:27:32.489475+0000","last_clean":"2024-09-17T05:27:32.489475+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:27:32.489475+0000","last_undegraded":"2024-09-17T05:27:32.489475+0000","last_fullsized":"2024-09-17T05:27:32.489475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490115+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:27:32.490115+0000","last_peered":"2024-09-17T05:27:32.490115+0000","last_clean":"2024-09-17T05:27:32.490115+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:27:32.490115+0000","last_undegraded":"2024-09-17T05:27:32.490115+0000","last_fullsized":"2024-09-17T05:27:32.490115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"25'11","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747548+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:27:48.747548+0000","last_peered":"2024-09-17T05:27:48.747548+0000","last_clean":"2024-09-17T05:27:48.747548+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:27:48.747548+0000","last_undegraded":"2024-09-17T05:27:48.747548+0000","last_fullsized":"2024-09-17T05:27:48.747548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688347+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:27:31.688347+0000","last_peered":"2024-09-17T05:27:31.688347+0000","last_clean":"2024-09-17T05:27:31.688347+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:27:31.688347+0000","last_undegraded":"2024-09-17T05:27:31.688347+0000","last_fullsized":"2024-09-17T05:27:31.688347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489012+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:27:32.489012+0000","last_peered":"2024-09-17T05:27:32.489012+0000","last_clean":"2024-09-17T05:27:32.489012+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:27:32.489012+0000","last_undegraded":"2024-09-17T05:27:32.489012+0000","last_fullsized":"2024-09-17T05:27:32.489012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"25'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488889+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:27:32.488889+0000","last_peered":"2024-09-17T05:27:32.488889+0000","last_clean":"2024-09-17T05:27:32.488889+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:27:32.488889+0000","last_undegraded":"2024-09-17T05:27:32.488889+0000","last_fullsized":"2024-09-17T05:27:32.488889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488867+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:27:32.488867+0000","last_peered":"2024-09-17T05:27:32.488867+0000","last_clean":"2024-09-17T05:27:32.488867+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:27:32.488867+0000","last_undegraded":"2024-09-17T05:27:32.488867+0000","last_fullsized":"2024-09-17T05:27:32.488867+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688333+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:27:31.688333+0000","last_peered":"2024-09-17T05:27:31.688333+0000","last_clean":"2024-09-17T05:27:31.688333+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:27:31.688333+0000","last_undegraded":"2024-09-17T05:27:31.688333+0000","last_fullsized":"2024-09-17T05:27:31.688333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.746302+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:27:48.746302+0000","last_peered":"2024-09-17T05:27:48.746302+0000","last_clean":"2024-09-17T05:27:48.746302+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:27:48.746302+0000","last_undegraded":"2024-09-17T05:27:48.746302+0000","last_fullsized":"2024-09-17T05:27:48.746302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488273+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:27:32.488273+0000","last_peered":"2024-09-17T05:27:32.488273+0000","last_clean":"2024-09-17T05:27:32.488273+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:27:32.488273+0000","last_undegraded":"2024-09-17T05:27:32.488273+0000","last_fullsized":"2024-09-17T05:27:32.488273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489164+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:27:32.489164+0000","last_peered":"2024-09-17T05:27:32.489164+0000","last_clean":"2024-09-17T05:27:32.489164+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:27:32.489164+0000","last_undegraded":"2024-09-17T05:27:32.489164+0000","last_fullsized":"2024-09-17T05:27:32.489164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.747034+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:27:48.747034+0000","last_peered":"2024-09-17T05:27:48.747034+0000","last_clean":"2024-09-17T05:27:48.747034+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:27:48.747034+0000","last_undegraded":"2024-09-17T05:27:48.747034+0000","last_fullsized":"2024-09-17T05:27:48.747034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688402+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:27:31.688402+0000","last_peered":"2024-09-17T05:27:31.688402+0000","last_clean":"2024-09-17T05:27:31.688402+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:27:31.688402+0000","last_undegraded":"2024-09-17T05:27:31.688402+0000","last_fullsized":"2024-09-17T05:27:31.688402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686347+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:27:31.686347+0000","last_peered":"2024-09-17T05:27:31.686347+0000","last_clean":"2024-09-17T05:27:31.686347+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:27:31.686347+0000","last_undegraded":"2024-09-17T05:27:31.686347+0000","last_fullsized":"2024-09-17T05:27:31.686347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688499+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:27:31.688499+0000","last_peered":"2024-09-17T05:27:31.688499+0000","last_clean":"2024-09-17T05:27:31.688499+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:27:31.688499+0000","last_undegraded":"2024-09-17T05:27:31.688499+0000","last_fullsized":"2024-09-17T05:27:31.688499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"25'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488921+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:27:32.488921+0000","last_peered":"2024-09-17T05:27:32.488921+0000","last_clean":"2024-09-17T05:27:32.488921+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:27:32.488921+0000","last_undegraded":"2024-09-17T05:27:32.488921+0000","last_fullsized":"2024-09-17T05:27:32.488921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490481+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:27:32.490481+0000","last_peered":"2024-09-17T05:27:32.490481+0000","last_clean":"2024-09-17T05:27:32.490481+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:27:32.490481+0000","last_undegraded":"2024-09-17T05:27:32.490481+0000","last_fullsized":"2024-09-17T05:27:32.490481+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686516+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:27:31.686516+0000","last_peered":"2024-09-17T05:27:31.686516+0000","last_clean":"2024-09-17T05:27:31.686516+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:27:31.686516+0000","last_undegraded":"2024-09-17T05:27:31.686516+0000","last_fullsized":"2024-09-17T05:27:31.686516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"25'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.771114+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:27:54.771114+0000","last_peered":"2024-09-17T05:27:54.771114+0000","last_clean":"2024-09-17T05:27:54.771114+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:27:54.771114+0000","last_undegraded":"2024-09-17T05:27:54.771114+0000","last_fullsized":"2024-09-17T05:27:54.771114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"25'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:54.767532+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:27:54.767532+0000","last_peered":"2024-09-17T05:27:54.767532+0000","last_clean":"2024-09-17T05:27:54.767532+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:27:54.767532+0000","last_undegraded":"2024-09-17T05:27:54.767532+0000","last_fullsized":"2024-09-17T05:27:54.767532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688804+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:27:31.688804+0000","last_peered":"2024-09-17T05:27:31.688804+0000","last_clean":"2024-09-17T05:27:31.688804+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:27:31.688804+0000","last_undegraded":"2024-09-17T05:27:31.688804+0000","last_fullsized":"2024-09-17T05:27:31.688804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688638+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:27:31.688638+0000","last_peered":"2024-09-17T05:27:31.688638+0000","last_clean":"2024-09-17T05:27:31.688638+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:27:31.688638+0000","last_undegraded":"2024-09-17T05:27:31.688638+0000","last_fullsized":"2024-09-17T05:27:31.688638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"25'6","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:48.745637+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:27:48.745637+0000","last_peered":"2024-09-17T05:27:48.745637+0000","last_clean":"2024-09-17T05:27:48.745637+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:27:48.745637+0000","last_undegraded":"2024-09-17T05:27:48.745637+0000","last_fullsized":"2024-09-17T05:27:48.745637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":878,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763479+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:27:31.763479+0000","last_peered":"2024-09-17T05:27:31.763479+0000","last_clean":"2024-09-17T05:27:31.763479+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:27:31.763479+0000","last_undegraded":"2024-09-17T05:27:31.763479+0000","last_fullsized":"2024-09-17T05:27:31.763479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488595+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:27:32.488595+0000","last_peered":"2024-09-17T05:27:32.488595+0000","last_clean":"2024-09-17T05:27:32.488595+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:27:32.488595+0000","last_undegraded":"2024-09-17T05:27:32.488595+0000","last_fullsized":"2024-09-17T05:27:32.488595+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488179+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:27:32.488179+0000","last_peered":"2024-09-17T05:27:32.488179+0000","last_clean":"2024-09-17T05:27:32.488179+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:27:32.488179+0000","last_undegraded":"2024-09-17T05:27:32.488179+0000","last_fullsized":"2024-09-17T05:27:32.488179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488202+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:27:32.488202+0000","last_peered":"2024-09-17T05:27:32.488202+0000","last_clean":"2024-09-17T05:27:32.488202+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:27:32.488202+0000","last_undegraded":"2024-09-17T05:27:32.488202+0000","last_fullsized":"2024-09-17T05:27:32.488202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490370+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:27:32.490370+0000","last_peered":"2024-09-17T05:27:32.490370+0000","last_clean":"2024-09-17T05:27:32.490370+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:27:32.490370+0000","last_undegraded":"2024-09-17T05:27:32.490370+0000","last_fullsized":"2024-09-17T05:27:32.490370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686893+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:27:31.686893+0000","last_peered":"2024-09-17T05:27:31.686893+0000","last_clean":"2024-09-17T05:27:31.686893+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:27:31.686893+0000","last_undegraded":"2024-09-17T05:27:31.686893+0000","last_fullsized":"2024-09-17T05:27:31.686893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687464+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:27:31.687464+0000","last_peered":"2024-09-17T05:27:31.687464+0000","last_clean":"2024-09-17T05:27:31.687464+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:27:31.687464+0000","last_undegraded":"2024-09-17T05:27:31.687464+0000","last_fullsized":"2024-09-17T05:27:31.687464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686068+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:27:31.686068+0000","last_peered":"2024-09-17T05:27:31.686068+0000","last_clean":"2024-09-17T05:27:31.686068+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:27:31.686068+0000","last_undegraded":"2024-09-17T05:27:31.686068+0000","last_fullsized":"2024-09-17T05:27:31.686068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489061+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:27:32.489061+0000","last_peered":"2024-09-17T05:27:32.489061+0000","last_clean":"2024-09-17T05:27:32.489061+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:27:32.489061+0000","last_undegraded":"2024-09-17T05:27:32.489061+0000","last_fullsized":"2024-09-17T05:27:32.489061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687936+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:27:31.687936+0000","last_peered":"2024-09-17T05:27:31.687936+0000","last_clean":"2024-09-17T05:27:31.687936+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:27:31.687936+0000","last_undegraded":"2024-09-17T05:27:31.687936+0000","last_fullsized":"2024-09-17T05:27:31.687936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488775+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:27:32.488775+0000","last_peered":"2024-09-17T05:27:32.488775+0000","last_clean":"2024-09-17T05:27:32.488775+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:27:32.488775+0000","last_undegraded":"2024-09-17T05:27:32.488775+0000","last_fullsized":"2024-09-17T05:27:32.488775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686014+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:27:31.686014+0000","last_peered":"2024-09-17T05:27:31.686014+0000","last_clean":"2024-09-17T05:27:31.686014+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:27:31.686014+0000","last_undegraded":"2024-09-17T05:27:31.686014+0000","last_fullsized":"2024-09-17T05:27:31.686014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.763450+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:27:31.763450+0000","last_peered":"2024-09-17T05:27:31.763450+0000","last_clean":"2024-09-17T05:27:31.763450+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:27:31.763450+0000","last_undegraded":"2024-09-17T05:27:31.763450+0000","last_fullsized":"2024-09-17T05:27:31.763450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688696+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:27:31.688696+0000","last_peered":"2024-09-17T05:27:31.688696+0000","last_clean":"2024-09-17T05:27:31.688696+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:27:31.688696+0000","last_undegraded":"2024-09-17T05:27:31.688696+0000","last_fullsized":"2024-09-17T05:27:31.688696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489945+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:27:32.489945+0000","last_peered":"2024-09-17T05:27:32.489945+0000","last_clean":"2024-09-17T05:27:32.489945+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:27:32.489945+0000","last_undegraded":"2024-09-17T05:27:32.489945+0000","last_fullsized":"2024-09-17T05:27:32.489945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490089+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:27:32.490089+0000","last_peered":"2024-09-17T05:27:32.490089+0000","last_clean":"2024-09-17T05:27:32.490089+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:27:32.490089+0000","last_undegraded":"2024-09-17T05:27:32.490089+0000","last_fullsized":"2024-09-17T05:27:32.490089+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488758+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:27:32.488758+0000","last_peered":"2024-09-17T05:27:32.488758+0000","last_clean":"2024-09-17T05:27:32.488758+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:27:32.488758+0000","last_undegraded":"2024-09-17T05:27:32.488758+0000","last_fullsized":"2024-09-17T05:27:32.488758+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490703+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:27:32.490703+0000","last_peered":"2024-09-17T05:27:32.490703+0000","last_clean":"2024-09-17T05:27:32.490703+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:27:32.490703+0000","last_undegraded":"2024-09-17T05:27:32.490703+0000","last_fullsized":"2024-09-17T05:27:32.490703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488669+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:27:32.488669+0000","last_peered":"2024-09-17T05:27:32.488669+0000","last_clean":"2024-09-17T05:27:32.488669+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:27:32.488669+0000","last_undegraded":"2024-09-17T05:27:32.488669+0000","last_fullsized":"2024-09-17T05:27:32.488669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.685843+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:27:31.685843+0000","last_peered":"2024-09-17T05:27:31.685843+0000","last_clean":"2024-09-17T05:27:31.685843+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:27:31.685843+0000","last_undegraded":"2024-09-17T05:27:31.685843+0000","last_fullsized":"2024-09-17T05:27:31.685843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686142+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:27:31.686142+0000","last_peered":"2024-09-17T05:27:31.686142+0000","last_clean":"2024-09-17T05:27:31.686142+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:27:31.686142+0000","last_undegraded":"2024-09-17T05:27:31.686142+0000","last_fullsized":"2024-09-17T05:27:31.686142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687059+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:27:31.687059+0000","last_peered":"2024-09-17T05:27:31.687059+0000","last_clean":"2024-09-17T05:27:31.687059+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:27:31.687059+0000","last_undegraded":"2024-09-17T05:27:31.687059+0000","last_fullsized":"2024-09-17T05:27:31.687059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489853+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:27:32.489853+0000","last_peered":"2024-09-17T05:27:32.489853+0000","last_clean":"2024-09-17T05:27:32.489853+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:27:32.489853+0000","last_undegraded":"2024-09-17T05:27:32.489853+0000","last_fullsized":"2024-09-17T05:27:32.489853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.490028+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:27:32.490028+0000","last_peered":"2024-09-17T05:27:32.490028+0000","last_clean":"2024-09-17T05:27:32.490028+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:27:32.490028+0000","last_undegraded":"2024-09-17T05:27:32.490028+0000","last_fullsized":"2024-09-17T05:27:32.490028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488438+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:27:32.488438+0000","last_peered":"2024-09-17T05:27:32.488438+0000","last_clean":"2024-09-17T05:27:32.488438+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:27:32.488438+0000","last_undegraded":"2024-09-17T05:27:32.488438+0000","last_fullsized":"2024-09-17T05:27:32.488438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.764205+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:27:31.764205+0000","last_peered":"2024-09-17T05:27:31.764205+0000","last_clean":"2024-09-17T05:27:31.764205+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:27:31.764205+0000","last_undegraded":"2024-09-17T05:27:31.764205+0000","last_fullsized":"2024-09-17T05:27:31.764205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.686254+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:27:31.686254+0000","last_peered":"2024-09-17T05:27:31.686254+0000","last_clean":"2024-09-17T05:27:31.686254+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:27:31.686254+0000","last_undegraded":"2024-09-17T05:27:31.686254+0000","last_fullsized":"2024-09-17T05:27:31.686254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488392+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:27:32.488392+0000","last_peered":"2024-09-17T05:27:32.488392+0000","last_clean":"2024-09-17T05:27:32.488392+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:27:32.488392+0000","last_undegraded":"2024-09-17T05:27:32.488392+0000","last_fullsized":"2024-09-17T05:27:32.488392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488190+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:27:32.488190+0000","last_peered":"2024-09-17T05:27:32.488190+0000","last_clean":"2024-09-17T05:27:32.488190+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:27:32.488190+0000","last_undegraded":"2024-09-17T05:27:32.488190+0000","last_fullsized":"2024-09-17T05:27:32.488190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489117+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:27:32.489117+0000","last_peered":"2024-09-17T05:27:32.489117+0000","last_clean":"2024-09-17T05:27:32.489117+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:27:32.489117+0000","last_undegraded":"2024-09-17T05:27:32.489117+0000","last_fullsized":"2024-09-17T05:27:32.489117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.488256+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:27:32.488256+0000","last_peered":"2024-09-17T05:27:32.488256+0000","last_clean":"2024-09-17T05:27:32.488256+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:27:32.488256+0000","last_undegraded":"2024-09-17T05:27:32.488256+0000","last_fullsized":"2024-09-17T05:27:32.488256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.687446+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:27:31.687446+0000","last_peered":"2024-09-17T05:27:31.687446+0000","last_clean":"2024-09-17T05:27:31.687446+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:27:31.687446+0000","last_undegraded":"2024-09-17T05:27:31.687446+0000","last_fullsized":"2024-09-17T05:27:31.687446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:31.688581+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:27:31.688581+0000","last_peered":"2024-09-17T05:27:31.688581+0000","last_clean":"2024-09-17T05:27:31.688581+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:27:31.688581+0000","last_undegraded":"2024-09-17T05:27:31.688581+0000","last_fullsized":"2024-09-17T05:27:31.688581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-17T05:27:32.489316+0000","last_change":"2024-09-17T05:27:26.662316+0000","last_active":"2024-09-17T05:27:32.489316+0000","last_peered":"2024-09-17T05:27:32.489316+0000","last_clean":"2024-09-17T05:27:32.489316+0000","last_became_active":"2024-09-17T05:27:26.662006+0000","last_became_peered":"2024-09-17T05:27:26.662006+0000","last_unstale":"2024-09-17T05:27:32.489316+0000","last_undegraded":"2024-09-17T05:27:32.489316+0000","last_fullsized":"2024-09-17T05:27:32.489316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:26.087311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7708,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":15416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":105939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":331776,"data_stored":692591,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":103379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":102971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":303104,"data_stored":107473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":693235,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:28:06.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 -- 172.21.15.117:0/2196668743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f804815b2f0 msgr2=0x7f80480d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2196668743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f804815b2f0 0x7f80480d0e30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f804806f390 tx=0x7f803802a5f0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 -- 172.21.15.117:0/2196668743 shutdown_connections 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2196668743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f804815b2f0 0x7f80480d0e30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2196668743 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80480c62c0 0x7f80480c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2196668743 >> v2:172.21.15.119:3300/0 conn(0x7f80480cb390 0x7f80480d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 -- 172.21.15.117:0/2196668743 >> 172.21.15.117:0/2196668743 conn(0x7f8048075e00 msgr2=0x7f804806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:06.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 -- 172.21.15.117:0/2196668743 shutdown_connections 2024-09-17T05:28:06.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.356+0000 7f804fcbe640 1 -- 172.21.15.117:0/2196668743 wait complete. 2024-09-17T05:28:06.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.358+0000 7f804fcbe640 1 Processor -- start 2024-09-17T05:28:06.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.358+0000 7f804fcbe640 1 -- start start 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804fcbe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 0x7f8048152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804fcbe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80480cb390 0x7f8048152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804fcbe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804815b2f0 0x7f8048154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804fcbe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f80480c5da0 con 0x7f80480c62c0 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804fcbe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f804815f6b0 con 0x7f804815b2f0 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804fcbe640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f80480c6c10 con 0x7f80480cb390 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 0x7f8048152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 0x7f8048152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40484/0 (socket says 172.21.15.117:40484) 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 -- 172.21.15.117:0/2703946946 learned_addr learned my addr 172.21.15.117:0/2703946946 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:06.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804e25b640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804815b2f0 0x7f8048154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804d259640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80480cb390 0x7f8048152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 -- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80480cb390 msgr2=0x7f8048152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80480cb390 0x7f8048152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 -- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804815b2f0 msgr2=0x7f8048154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804815b2f0 0x7f8048154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 -- 172.21.15.117:0/2703946946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8048153070 con 0x7f80480c62c0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.359+0000 7f804da5a640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 0x7f8048152b30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f804400d780 tx=0x7f804400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.360+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044017070 con 0x7f80480c62c0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.360+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8044004310 con 0x7f80480c62c0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.360+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804815c700 con 0x7f80480c62c0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.360+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044005280 con 0x7f80480c62c0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.360+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8048158600 con 0x7f80480c62c0 2024-09-17T05:28:06.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.361+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80481f2520 con 0x7f80480c62c0 2024-09-17T05:28:06.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.361+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8044005420 con 0x7f80480c62c0 2024-09-17T05:28:06.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.362+0000 7f803e7fc640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f802c078bc0 0x7f802c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.362+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f804409b2c0 con 0x7f80480c62c0 2024-09-17T05:28:06.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.362+0000 7f804d259640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f802c078bc0 0x7f802c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.363+0000 7f804d259640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f802c078bc0 0x7f802c07b0d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8038033000 tx=0x7f8038004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:06.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.366+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8044020020 con 0x7f80480c62c0 2024-09-17T05:28:06.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.578+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80480c7950 con 0x7f80480c62c0 2024-09-17T05:28:06.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.579+0000 7f803e7fc640 1 -- 172.21.15.117:0/2703946946 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7f804409b0e0 con 0x7f80480c62c0 2024-09-17T05:28:06.580 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:06.581 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:28:06.581 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f802c078bc0 msgr2=0x7f802c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f802c078bc0 0x7f802c07b0d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8038033000 tx=0x7f8038004730 comp rx=0 tx=0).stop 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 msgr2=0x7f8048152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 0x7f8048152b30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f804400d780 tx=0x7f804400dc40 comp rx=0 tx=0).stop 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 shutdown_connections 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f802c078bc0 0x7f802c07b0d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804815b2f0 0x7f8048154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80480cb390 0x7f8048152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 --2- 172.21.15.117:0/2703946946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80480c62c0 0x7f8048152b30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 >> 172.21.15.117:0/2703946946 conn(0x7f8048075e00 msgr2=0x7f8048074c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:06.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 shutdown_connections 2024-09-17T05:28:06.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.581+0000 7f804fcbe640 1 -- 172.21.15.117:0/2703946946 wait complete. 2024-09-17T05:28:06.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:28:06.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 -- 172.21.15.117:0/3545100916 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40c69d0 msgr2=0x7f77e40d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 --2- 172.21.15.117:0/3545100916 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40c69d0 0x7f77e40d0e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77dc005fd0 tx=0x7f77dc02a940 comp rx=0 tx=0).stop 2024-09-17T05:28:06.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 -- 172.21.15.117:0/3545100916 shutdown_connections 2024-09-17T05:28:06.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 --2- 172.21.15.117:0/3545100916 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40c69d0 0x7f77e40d0e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 --2- 172.21.15.117:0/3545100916 >> v2:172.21.15.119:3300/0 conn(0x7f77e40cb390 0x7f77e40d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 -- 172.21.15.117:0/3545100916 >> 172.21.15.117:0/3545100916 conn(0x7f77e4075e00 msgr2=0x7f77e406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:06.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 -- 172.21.15.117:0/3545100916 shutdown_connections 2024-09-17T05:28:06.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.707+0000 7f77eb6be640 1 -- 172.21.15.117:0/3545100916 wait complete. 2024-09-17T05:28:06.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.709+0000 7f77eb6be640 1 Processor -- start 2024-09-17T05:28:06.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.709+0000 7f77eb6be640 1 -- start start 2024-09-17T05:28:06.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77eb6be640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77e40c69d0 0x7f77e4157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77eb6be640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40cb390 0x7f77e40cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77eb6be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 0x7f77e40ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77eb6be640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f77e40c5da0 con 0x7f77e415b2f0 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77eb6be640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f77e415f6b0 con 0x7f77e40c69d0 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77eb6be640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f77e41558b0 con 0x7f77e40cb390 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77e9c5b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 0x7f77e40ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77e9c5b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 0x7f77e40ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40498/0 (socket says 172.21.15.117:40498) 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77e9c5b640 1 -- 172.21.15.117:0/578989330 learned_addr learned my addr 172.21.15.117:0/578989330 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77e945a640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77e40c69d0 0x7f77e4157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.710+0000 7f77e8c59640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40cb390 0x7f77e40cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77e9c5b640 1 -- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40cb390 msgr2=0x7f77e40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77e9c5b640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40cb390 0x7f77e40cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77e9c5b640 1 -- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77e40c69d0 msgr2=0x7f77e4157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77e9c5b640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77e40c69d0 0x7f77e4157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77e9c5b640 1 -- 172.21.15.117:0/578989330 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e41578c0 con 0x7f77e415b2f0 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77e9c5b640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 0x7f77e40ca400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009cd0 tx=0x7f77dc031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc03e070 con 0x7f77e415b2f0 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77dc004540 con 0x7f77e415b2f0 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e406eca0 con 0x7f77e415b2f0 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc030430 con 0x7f77e415b2f0 2024-09-17T05:28:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.711+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e41552a0 con 0x7f77e415b2f0 2024-09-17T05:28:06.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.712+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e40c7950 con 0x7f77e415b2f0 2024-09-17T05:28:06.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.713+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f77dc043460 con 0x7f77e415b2f0 2024-09-17T05:28:06.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.713+0000 7f77d9ffb640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f77c8078b90 0x7f77c807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:06.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.713+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6590+0+0 (secure 0 0 0) 0x7f77dc0bc3f0 con 0x7f77e415b2f0 2024-09-17T05:28:06.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.714+0000 7f77e945a640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f77c8078b90 0x7f77c807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:06.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.714+0000 7f77e945a640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f77c8078b90 0x7f77c807b0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f77e00095f0 tx=0x7f77e0005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:06.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.718+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f77dc04a070 con 0x7f77e415b2f0 2024-09-17T05:28:06.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.895+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77e415c080 con 0x7f77e415b2f0 2024-09-17T05:28:06.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.896+0000 7f77d9ffb640 1 -- 172.21.15.117:0/578989330 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18769 (secure 0 0 0) 0x7f77dc083e80 con 0x7f77e415b2f0 2024-09-17T05:28:06.898 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:06.898 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":27,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:28:06.816642+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3f","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:28:06.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f77c8078b90 msgr2=0x7f77c807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f77c8078b90 0x7f77c807b0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f77e00095f0 tx=0x7f77e0005f90 comp rx=0 tx=0).stop 2024-09-17T05:28:06.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 msgr2=0x7f77e40ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 0x7f77e40ca400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009cd0 tx=0x7f77dc031e20 comp rx=0 tx=0).stop 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 shutdown_connections 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f77c8078b90 0x7f77c807b0a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e415b2f0 0x7f77e40ca400 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f77e40cb390 0x7f77e40cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 --2- 172.21.15.117:0/578989330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77e40c69d0 0x7f77e4157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 >> 172.21.15.117:0/578989330 conn(0x7f77e4075e00 msgr2=0x7f77e40b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 shutdown_connections 2024-09-17T05:28:06.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:06.899+0000 7f77eb6be640 1 -- 172.21.15.117:0/578989330 wait complete. 2024-09-17T05:28:06.912 DEBUG:tasks.mds_thrash:Yielding 2024-09-17T05:28:06.912 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T05:28:06.928 INFO:tasks.workunit:Pulling workunits from ref ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:28:06.928 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T05:28:06.928 INFO:tasks.workunit:timeout=3h 2024-09-17T05:28:06.928 INFO:tasks.workunit:cleanup=True 2024-09-17T05:28:06.929 DEBUG:teuthology.orchestra.run.smithi117:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:06.929 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-09-17T05:28:06.929 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 50 secs before thrashing 2024-09-17T05:28:06.944 INFO:teuthology.orchestra.run.smithi117.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 29h/41d Inode: 1 Links: 2 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-17 05:27:31.638109109 +0000 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-17 05:27:40.248818569 +0000 2024-09-17T05:28:06.945 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-17 05:27:31.638109109 +0000 2024-09-17T05:28:06.945 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-17T05:28:06.945 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-17T05:28:07.019 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T05:28:07.075 INFO:tasks.workunit.client.0.smithi117.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T05:28:22.193 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:28:22.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:28:52.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:28:52.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:28:56.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:28:57.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.104+0000 7f9814769640 1 -- 172.21.15.117:0/1147716269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 msgr2=0x7f980c0c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.104+0000 7f9814769640 1 --2- 172.21.15.117:0/1147716269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c0c9730 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9804009c60 tx=0x7f980402a960 comp rx=0 tx=0).stop 2024-09-17T05:28:57.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 -- 172.21.15.117:0/1147716269 shutdown_connections 2024-09-17T05:28:57.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 --2- 172.21.15.117:0/1147716269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c0c9730 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 --2- 172.21.15.117:0/1147716269 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f980c153260 0x7f980c15d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 --2- 172.21.15.117:0/1147716269 >> v2:172.21.15.119:3300/0 conn(0x7f980c0cad30 0x7f980c0c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 -- 172.21.15.117:0/1147716269 >> 172.21.15.117:0/1147716269 conn(0x7f980c076430 msgr2=0x7f980c072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:57.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 -- 172.21.15.117:0/1147716269 shutdown_connections 2024-09-17T05:28:57.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.105+0000 7f9814769640 1 -- 172.21.15.117:0/1147716269 wait complete. 2024-09-17T05:28:57.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.108+0000 7f9814769640 1 Processor -- start 2024-09-17T05:28:57.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.108+0000 7f9814769640 1 -- start start 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9814769640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f980c0cad30 0x7f980c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9814769640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f980c153260 0x7f980c1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9814769640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9814769640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f980c159f40 con 0x7f980c159b30 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9814769640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f980c0cbc30 con 0x7f980c0cad30 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9812d06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9811d04640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f980c153260 0x7f980c1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9812d06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39664/0 (socket says 172.21.15.117:39664) 2024-09-17T05:28:57.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9812d06640 1 -- 172.21.15.117:0/1010651131 learned_addr learned my addr 172.21.15.117:0/1010651131 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9812505640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f980c0cad30 0x7f980c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.111+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f980c1f3670 con 0x7f980c153260 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9812d06640 1 -- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f980c153260 msgr2=0x7f980c1519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9812d06640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f980c153260 0x7f980c1519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9812d06640 1 -- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f980c0cad30 msgr2=0x7f980c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9812d06640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f980c0cad30 0x7f980c0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9812d06640 1 -- 172.21.15.117:0/1010651131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980c15a600 con 0x7f980c159b30 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9812d06640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c151ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9804033040 tx=0x7f9804004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804034040 con 0x7f980c159b30 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9804004670 con 0x7f980c159b30 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98040092a0 con 0x7f980c159b30 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.112+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980c15a880 con 0x7f980c159b30 2024-09-17T05:28:57.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.113+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f980c1efd00 con 0x7f980c159b30 2024-09-17T05:28:57.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.114+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f980c15aa60 con 0x7f980c159b30 2024-09-17T05:28:57.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.115+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9804037020 con 0x7f980c159b30 2024-09-17T05:28:57.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.116+0000 7f9802ffd640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f97f007cfd0 0x7f97f007f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.116+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7f98040bc780 con 0x7f980c159b30 2024-09-17T05:28:57.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.116+0000 7f9812505640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f97f007cfd0 0x7f97f007f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.117+0000 7f9812505640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f97f007cfd0 0x7f97f007f4e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9808009ab0 tx=0x7f9808005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:57.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.125+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f980c15a880 con 0x7f980c159b30 2024-09-17T05:28:57.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.561+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f980c156a60 con 0x7f980c159b30 2024-09-17T05:28:57.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.561+0000 7f9802ffd640 1 -- 172.21.15.117:0/1010651131 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7f980c15a600 con 0x7f980c159b30 2024-09-17T05:28:57.563 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:57.563 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:28:57.563 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:28:57.566 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f97f007cfd0 msgr2=0x7f97f007f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f97f007cfd0 0x7f97f007f4e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9808009ab0 tx=0x7f9808005f90 comp rx=0 tx=0).stop 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 msgr2=0x7f980c151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c151ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9804033040 tx=0x7f9804004270 comp rx=0 tx=0).stop 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 shutdown_connections 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f97f007cfd0 0x7f97f007f4e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f980c159b30 0x7f980c151ef0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f980c153260 0x7f980c1519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 --2- 172.21.15.117:0/1010651131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f980c0cad30 0x7f980c0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.564+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 >> 172.21.15.117:0/1010651131 conn(0x7f980c076430 msgr2=0x7f980c1495a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.565+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 shutdown_connections 2024-09-17T05:28:57.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.565+0000 7f9814769640 1 -- 172.21.15.117:0/1010651131 wait complete. 2024-09-17T05:28:57.581 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2024-09-17T05:28:57.581 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-09-17T05:28:57.581 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:28:57.592 INFO:tasks.ceph.mds.e:Stopped 2024-09-17T05:28:57.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:28:57.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 -- 172.21.15.117:0/662219873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff808152300 msgr2=0x7ff808158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/662219873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff808152300 0x7ff808158a10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff8080760e0 tx=0x7ff7f802a680 comp rx=0 tx=0).stop 2024-09-17T05:28:57.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 -- 172.21.15.117:0/662219873 shutdown_connections 2024-09-17T05:28:57.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/662219873 >> v2:172.21.15.119:3300/0 conn(0x7ff80806e2c0 0x7ff8080741f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/662219873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff808152300 0x7ff808158a10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 -- 172.21.15.117:0/662219873 >> 172.21.15.117:0/662219873 conn(0x7ff8080c9a20 msgr2=0x7ff8080c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:57.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 -- 172.21.15.117:0/662219873 shutdown_connections 2024-09-17T05:28:57.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.804+0000 7ff80d4d5640 1 -- 172.21.15.117:0/662219873 wait complete. 2024-09-17T05:28:57.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.807+0000 7ff80d4d5640 1 Processor -- start 2024-09-17T05:28:57.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.808+0000 7ff80d4d5640 1 -- start start 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff80d4d5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff80806deb0 0x7ff8080c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff80d4d5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 0x7ff808154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff80d4d5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff808152300 0x7ff8081f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff80d4d5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff80815c1a0 con 0x7ff80806e2c0 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff80d4d5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff80815bec0 con 0x7ff808152300 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff80d4d5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff8080c48d0 con 0x7ff80806deb0 2024-09-17T05:28:57.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff8067fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 0x7ff808154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff8067fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 0x7ff808154940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39674/0 (socket says 172.21.15.117:39674) 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff8067fc640 1 -- 172.21.15.117:0/3702460186 learned_addr learned my addr 172.21.15.117:0/3702460186 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff8077fe640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff808152300 0x7ff8081f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.810+0000 7ff806ffd640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff80806deb0 0x7ff8080c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff8067fc640 1 -- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff80806deb0 msgr2=0x7ff8080c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff8067fc640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff80806deb0 0x7ff8080c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff8067fc640 1 -- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff808152300 msgr2=0x7ff8081f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff8067fc640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff808152300 0x7ff8081f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff8067fc640 1 -- 172.21.15.117:0/3702460186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8081f3b40 con 0x7ff80806e2c0 2024-09-17T05:28:57.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff806ffd640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff80806deb0 0x7ff8080c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:28:57.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff8067fc640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 0x7ff808154940 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c370 tx=0x7ff7fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:57.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc017070 con 0x7ff80806e2c0 2024-09-17T05:28:57.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8081f3e20 con 0x7ff80806e2c0 2024-09-17T05:28:57.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7fc0053c0 con 0x7ff80806e2c0 2024-09-17T05:28:57.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.811+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc017070 con 0x7ff80806e2c0 2024-09-17T05:28:57.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.812+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8080c1630 con 0x7ff80806e2c0 2024-09-17T05:28:57.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.814+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff808158a10 con 0x7ff80806e2c0 2024-09-17T05:28:57.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.814+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff7fc01a070 con 0x7ff80806e2c0 2024-09-17T05:28:57.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.815+0000 7ff7ef7fe640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7f4078d90 0x7ff7f407b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:28:57.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.815+0000 7ff806ffd640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7f4078d90 0x7ff7f407b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:28:57.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.816+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7ff7fc097fc0 con 0x7ff80806e2c0 2024-09-17T05:28:57.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.816+0000 7ff806ffd640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7f4078d90 0x7ff7f407b2a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8033000 tx=0x7ff7f8004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:28:57.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:57.824+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff7fc0a2020 con 0x7ff80806e2c0 2024-09-17T05:28:58.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-09-17T05:28:58.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.262+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8080c13c0 con 0x7ff80806e2c0 2024-09-17T05:28:58.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.263+0000 7ff7ef7fe640 1 -- 172.21.15.117:0/3702460186 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7ff808158f50 con 0x7ff80806e2c0 2024-09-17T05:28:58.264 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:28:58.265 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:28:58.265 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:28:58.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7f4078d90 msgr2=0x7ff7f407b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:58.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7f4078d90 0x7ff7f407b2a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff7f8033000 tx=0x7ff7f8004730 comp rx=0 tx=0).stop 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 msgr2=0x7ff808154940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 0x7ff808154940 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c370 tx=0x7ff7fc00c830 comp rx=0 tx=0).stop 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 shutdown_connections 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7f4078d90 0x7ff7f407b2a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff808152300 0x7ff8081f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff80806e2c0 0x7ff808154940 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 --2- 172.21.15.117:0/3702460186 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff80806deb0 0x7ff8080c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 >> 172.21.15.117:0/3702460186 conn(0x7ff8080c9a20 msgr2=0x7ff80814a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 shutdown_connections 2024-09-17T05:28:58.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:28:58.265+0000 7ff80d4d5640 1 -- 172.21.15.117:0/3702460186 wait complete. 2024-09-17T05:28:58.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-09-17T05:29:00.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.394+0000 7fed1de4d640 1 -- 172.21.15.117:0/2710366593 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180c9e60 msgr2=0x7fed180c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.394+0000 7fed1de4d640 1 --2- 172.21.15.117:0/2710366593 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180c9e60 0x7fed180c3540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009860 tx=0x7fed0c02a510 comp rx=0 tx=0).stop 2024-09-17T05:29:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.395+0000 7fed1de4d640 1 -- 172.21.15.117:0/2710366593 shutdown_connections 2024-09-17T05:29:00.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.395+0000 7fed1de4d640 1 --2- 172.21.15.117:0/2710366593 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180c9e60 0x7fed180c3540 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.395+0000 7fed1de4d640 1 -- 172.21.15.117:0/2710366593 >> 172.21.15.117:0/2710366593 conn(0x7fed1806cda0 msgr2=0x7fed1806d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:00.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.395+0000 7fed1de4d640 1 -- 172.21.15.117:0/2710366593 shutdown_connections 2024-09-17T05:29:00.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.395+0000 7fed1de4d640 1 -- 172.21.15.117:0/2710366593 wait complete. 2024-09-17T05:29:00.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.396+0000 7fed1de4d640 1 Processor -- start 2024-09-17T05:29:00.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.397+0000 7fed1de4d640 1 -- start start 2024-09-17T05:29:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed1de4d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 0x7fed18157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed1de4d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed181530e0 0x7fed1814c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed1de4d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180ce140 0x7fed181f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed1de4d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed180c3a80 con 0x7fed180c9e60 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed1de4d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fed18159540 con 0x7fed181530e0 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed1de4d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fed180cad80 con 0x7fed180ce140 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 0x7fed18157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 0x7fed18157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39678/0 (socket says 172.21.15.117:39678) 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 -- 172.21.15.117:0/937032278 learned_addr learned my addr 172.21.15.117:0/937032278 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed16ffd640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed181530e0 0x7fed1814c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed17fff640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180ce140 0x7fed181f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 -- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180ce140 msgr2=0x7fed181f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180ce140 0x7fed181f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 -- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed181530e0 msgr2=0x7fed1814c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed181530e0 0x7fed1814c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.398+0000 7fed177fe640 1 -- 172.21.15.117:0/937032278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18158100 con 0x7fed180c9e60 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.399+0000 7fed177fe640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 0x7fed18157bc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fed0800c460 tx=0x7fed0800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.399+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08016020 con 0x7fed180c9e60 2024-09-17T05:29:00.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.399+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed08012940 con 0x7fed180c9e60 2024-09-17T05:29:00.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.399+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08012d90 con 0x7fed180c9e60 2024-09-17T05:29:00.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.399+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed180cd6b0 con 0x7fed180c9e60 2024-09-17T05:29:00.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.399+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed181573c0 con 0x7fed180c9e60 2024-09-17T05:29:00.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.400+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed18077190 con 0x7fed180c9e60 2024-09-17T05:29:00.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.401+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fed08019020 con 0x7fed180c9e60 2024-09-17T05:29:00.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.402+0000 7fecfffff640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fed04078b90 0x7fed0407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:00.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.402+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7fed0809aeb0 con 0x7fed180c9e60 2024-09-17T05:29:00.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.402+0000 7fed16ffd640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fed04078b90 0x7fed0407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:00.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.403+0000 7fed16ffd640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fed04078b90 0x7fed0407b0a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed0c034000 tx=0x7fed0c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:00.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.410+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fed18077190 con 0x7fed180c9e60 2024-09-17T05:29:00.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.665+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed180ced80 con 0x7fed180c9e60 2024-09-17T05:29:00.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.666+0000 7fecfffff640 1 -- 172.21.15.117:0/937032278 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7fed08020020 con 0x7fed180c9e60 2024-09-17T05:29:00.668 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:00.668 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:29:00.668 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fed04078b90 msgr2=0x7fed0407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fed04078b90 0x7fed0407b0a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed0c034000 tx=0x7fed0c004730 comp rx=0 tx=0).stop 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 msgr2=0x7fed18157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 0x7fed18157bc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fed0800c460 tx=0x7fed0800c920 comp rx=0 tx=0).stop 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 shutdown_connections 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fed04078b90 0x7fed0407b0a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed180ce140 0x7fed181f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed181530e0 0x7fed1814c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 --2- 172.21.15.117:0/937032278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed180c9e60 0x7fed18157bc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 >> 172.21.15.117:0/937032278 conn(0x7fed1806cda0 msgr2=0x7fed18148110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:00.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 shutdown_connections 2024-09-17T05:29:00.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:00.669+0000 7fed1de4d640 1 -- 172.21.15.117:0/937032278 wait complete. 2024-09-17T05:29:00.682 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-09-17T05:29:02.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:02.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 -- 172.21.15.117:0/4205039601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8159d40 msgr2=0x7fdfa815cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:02.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 --2- 172.21.15.117:0/4205039601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8159d40 0x7fdfa815cfa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009830 tx=0x7fdf9802a530 comp rx=0 tx=0).stop 2024-09-17T05:29:02.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 -- 172.21.15.117:0/4205039601 shutdown_connections 2024-09-17T05:29:02.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 --2- 172.21.15.117:0/4205039601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8159d40 0x7fdfa815cfa0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:02.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 -- 172.21.15.117:0/4205039601 >> 172.21.15.117:0/4205039601 conn(0x7fdfa806cda0 msgr2=0x7fdfa806d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:02.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 -- 172.21.15.117:0/4205039601 shutdown_connections 2024-09-17T05:29:02.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.799+0000 7fdfac9da640 1 -- 172.21.15.117:0/4205039601 wait complete. 2024-09-17T05:29:02.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.801+0000 7fdfac9da640 1 Processor -- start 2024-09-17T05:29:02.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.801+0000 7fdfac9da640 1 -- start start 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfac9da640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfa80c9e60 0x7fdfa80c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfac9da640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfa81530e0 0x7fdfa815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfac9da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 0x7fdfa8147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfac9da640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8159540 con 0x7fdfa8146f60 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfac9da640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8159260 con 0x7fdfa81530e0 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfac9da640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa80cad80 con 0x7fdfa80c9e60 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfa6d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 0x7fdfa8147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfa6d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 0x7fdfa8147370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39690/0 (socket says 172.21.15.117:39690) 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfa6d9d640 1 -- 172.21.15.117:0/3149915725 learned_addr learned my addr 172.21.15.117:0/3149915725 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.802+0000 7fdfa659c640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfa80c9e60 0x7fdfa80c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:02.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa5d9b640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfa81530e0 0x7fdfa815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa6d9d640 1 -- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfa80c9e60 msgr2=0x7fdfa80c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa6d9d640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfa80c9e60 0x7fdfa80c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa6d9d640 1 -- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfa81530e0 msgr2=0x7fdfa815bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa6d9d640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfa81530e0 0x7fdfa815bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa6d9d640 1 -- 172.21.15.117:0/3149915725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa8155f70 con 0x7fdfa8146f60 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfa6d9d640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 0x7fdfa8147370 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000cae0 tx=0x7fdfa000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0016020 con 0x7fdfa8146f60 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfa000ad30 con 0x7fdfa8146f60 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfac9da640 1 -- 172.21.15.117:0/3149915725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa81555c0 con 0x7fdfa8146f60 2024-09-17T05:29:02.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0012680 con 0x7fdfa8146f60 2024-09-17T05:29:02.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.803+0000 7fdfac9da640 1 -- 172.21.15.117:0/3149915725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa8154d50 con 0x7fdfa8146f60 2024-09-17T05:29:02.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.805+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa81534f0 con 0x7fdfa8146f60 2024-09-17T05:29:02.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.806+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdfa0019020 con 0x7fdfa8146f60 2024-09-17T05:29:02.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.807+0000 7fdf96ffd640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdf88078b90 0x7fdf8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:02.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.807+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7fdfa009b3e0 con 0x7fdfa8146f60 2024-09-17T05:29:02.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.807+0000 7fdfa659c640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdf88078b90 0x7fdf8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:02.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.808+0000 7fdfa659c640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdf88078b90 0x7fdf8807b0a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdf98034000 tx=0x7fdf98004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:02.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:02.812+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdfa0020030 con 0x7fdfa8146f60 2024-09-17T05:29:03.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.031+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfa80ca950 con 0x7fdfa8146f60 2024-09-17T05:29:03.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.032+0000 7fdf96ffd640 1 -- 172.21.15.117:0/3149915725 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7fdfa0063550 con 0x7fdfa8146f60 2024-09-17T05:29:03.034 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:03.034 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:29:03.034 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:29:03.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdf88078b90 msgr2=0x7fdf8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdf88078b90 0x7fdf8807b0a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdf98034000 tx=0x7fdf98004730 comp rx=0 tx=0).stop 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 msgr2=0x7fdfa8147370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 0x7fdfa8147370 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000cae0 tx=0x7fdfa000a060 comp rx=0 tx=0).stop 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 shutdown_connections 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdf88078b90 0x7fdf8807b0a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfa8146f60 0x7fdfa8147370 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdfa81530e0 0x7fdfa815bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 --2- 172.21.15.117:0/3149915725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdfa80c9e60 0x7fdfa80c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 >> 172.21.15.117:0/3149915725 conn(0x7fdfa806cda0 msgr2=0x7fdfa814f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 shutdown_connections 2024-09-17T05:29:03.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:03.035+0000 7fdf94ff9640 1 -- 172.21.15.117:0/3149915725 wait complete. 2024-09-17T05:29:03.048 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-09-17T05:29:04.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e has been restored 2024-09-17T05:29:05.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:05.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 -- 172.21.15.117:0/562611940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 msgr2=0x7fb97c0c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:05.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 --2- 172.21.15.117:0/562611940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c0c3540 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb964005fd0 tx=0x7fb96402a5f0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 -- 172.21.15.117:0/562611940 shutdown_connections 2024-09-17T05:29:05.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 --2- 172.21.15.117:0/562611940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c0c3540 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 -- 172.21.15.117:0/562611940 >> 172.21.15.117:0/562611940 conn(0x7fb97c06cda0 msgr2=0x7fb97c06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:05.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 -- 172.21.15.117:0/562611940 shutdown_connections 2024-09-17T05:29:05.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.167+0000 7fb982313640 1 -- 172.21.15.117:0/562611940 wait complete. 2024-09-17T05:29:05.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.168+0000 7fb982313640 1 Processor -- start 2024-09-17T05:29:05.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.169+0000 7fb982313640 1 -- start start 2024-09-17T05:29:05.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb982313640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:05.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb982313640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb97c1530e0 0x7fb97c1ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:05.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb982313640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb97c1e5100 0x7fb97c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb982313640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c0c3a80 con 0x7fb97c0c9e60 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb982313640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c159540 con 0x7fb97c1e5100 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb982313640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb97c0cad80 con 0x7fb97c1530e0 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c14d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47868/0 (socket says 172.21.15.117:47868) 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 -- 172.21.15.117:0/1992127681 learned_addr learned my addr 172.21.15.117:0/1992127681 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb9808b0640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb97c1e5100 0x7fb97c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97b7fe640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb97c1530e0 0x7fb97c1ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:05.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 -- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb97c1530e0 msgr2=0x7fb97c1ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb97c1530e0 0x7fb97c1ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 -- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb97c1e5100 msgr2=0x7fb97c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb97c1e5100 0x7fb97c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97bfff640 1 -- 172.21.15.117:0/1992127681 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c1e2460 con 0x7fb97c0c9e60 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb9808b0640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb97c1e5100 0x7fb97c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.170+0000 7fb97b7fe640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb97c1530e0 0x7fb97c1ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.171+0000 7fb97bfff640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c14d410 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00c430 tx=0x7fb96c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.171+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c017070 con 0x7fb97c0c9e60 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.171+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb96c0053c0 con 0x7fb97c0c9e60 2024-09-17T05:29:05.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.171+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97c1efe50 con 0x7fb97c0c9e60 2024-09-17T05:29:05.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.171+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c003850 con 0x7fb97c0c9e60 2024-09-17T05:29:05.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.171+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97c1e29b0 con 0x7fb97c0c9e60 2024-09-17T05:29:05.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.172+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97c1534f0 con 0x7fb97c0c9e60 2024-09-17T05:29:05.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.173+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb96c01a070 con 0x7fb97c0c9e60 2024-09-17T05:29:05.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.173+0000 7fb978ff9640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb95c078b90 0x7fb95c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:05.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.174+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7fb96c09a480 con 0x7fb97c0c9e60 2024-09-17T05:29:05.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.174+0000 7fb97b7fe640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb95c078b90 0x7fb95c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:05.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.174+0000 7fb97b7fe640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb95c078b90 0x7fb95c07b0a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb964033000 tx=0x7fb964004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:05.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.178+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb96c0625e0 con 0x7fb97c0c9e60 2024-09-17T05:29:05.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.397+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb97c1ef670 con 0x7fb97c0c9e60 2024-09-17T05:29:05.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.398+0000 7fb978ff9640 1 -- 172.21.15.117:0/1992127681 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7fb96c061ee0 con 0x7fb97c0c9e60 2024-09-17T05:29:05.399 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:05.399 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:29:05.400 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb95c078b90 msgr2=0x7fb95c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb95c078b90 0x7fb95c07b0a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb964033000 tx=0x7fb964004730 comp rx=0 tx=0).stop 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 msgr2=0x7fb97c14d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c14d410 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00c430 tx=0x7fb96c00c8f0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 shutdown_connections 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb95c078b90 0x7fb95c07b0a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb97c1e5100 0x7fb97c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb97c1530e0 0x7fb97c1ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 --2- 172.21.15.117:0/1992127681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb97c0c9e60 0x7fb97c14d410 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:05.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 >> 172.21.15.117:0/1992127681 conn(0x7fb97c06cda0 msgr2=0x7fb97c157880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:05.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 shutdown_connections 2024-09-17T05:29:05.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:05.400+0000 7fb982313640 1 -- 172.21.15.117:0/1992127681 wait complete. 2024-09-17T05:29:05.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-09-17T05:29:07.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:07.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 -- 172.21.15.117:0/1334991884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 msgr2=0x7f7050071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:07.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1334991884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050071c60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7044005fd0 tx=0x7f704402a4a0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 -- 172.21.15.117:0/1334991884 shutdown_connections 2024-09-17T05:29:07.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1334991884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050071c60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1334991884 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f70500c67a0 0x7f7050152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 -- 172.21.15.117:0/1334991884 >> 172.21.15.117:0/1334991884 conn(0x7f705006e590 msgr2=0x7f70500cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:07.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 -- 172.21.15.117:0/1334991884 shutdown_connections 2024-09-17T05:29:07.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.529+0000 7f7054ecc640 1 -- 172.21.15.117:0/1334991884 wait complete. 2024-09-17T05:29:07.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.531+0000 7f7054ecc640 1 Processor -- start 2024-09-17T05:29:07.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.531+0000 7f7054ecc640 1 -- start start 2024-09-17T05:29:07.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.532+0000 7f7054ecc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:07.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70500c4f50 0x7f70500ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:07.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f70500c67a0 0x7f70501f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:07.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f70500c7ff0 con 0x7f7050071850 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f70500ce0d0 con 0x7f70500c4f50 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f70500c83a0 con 0x7f70500c67a0 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47878/0 (socket says 172.21.15.117:47878) 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 -- 172.21.15.117:0/1894014237 learned_addr learned my addr 172.21.15.117:0/1894014237 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704ed9d640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f70500c67a0 0x7f70501f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704dd9b640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70500c4f50 0x7f70500ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 -- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f70500c67a0 msgr2=0x7f70501f5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f70500c67a0 0x7f70501f5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 -- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70500c4f50 msgr2=0x7f70500ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70500c4f50 0x7f70500ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 -- 172.21.15.117:0/1894014237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70501f5cb0 con 0x7f7050071850 2024-09-17T05:29:07.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704dd9b640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70500c4f50 0x7f70500ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:29:07.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f704e59c640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050158e30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f704000c9a0 tx=0x7f704000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:07.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040007bb0 con 0x7f7050071850 2024-09-17T05:29:07.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7040007e40 con 0x7f7050071850 2024-09-17T05:29:07.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70500cbc90 con 0x7f7050071850 2024-09-17T05:29:07.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040005280 con 0x7f7050071850 2024-09-17T05:29:07.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.533+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70501f5ec0 con 0x7f7050071850 2024-09-17T05:29:07.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.534+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70500c7940 con 0x7f7050071850 2024-09-17T05:29:07.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.535+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7040005420 con 0x7f7050071850 2024-09-17T05:29:07.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.536+0000 7f703effd640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7030078b90 0x7f703007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:07.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.536+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7f704009b4d0 con 0x7f7050071850 2024-09-17T05:29:07.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.536+0000 7f704dd9b640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7030078b90 0x7f703007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:07.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.537+0000 7f704dd9b640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7030078b90 0x7f703007b0a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7044033000 tx=0x7f7044004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:07.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.545+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7040014030 con 0x7f7050071850 2024-09-17T05:29:07.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.762+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7050155a70 con 0x7f7050071850 2024-09-17T05:29:07.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.763+0000 7f703effd640 1 -- 172.21.15.117:0/1894014237 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7f7040063700 con 0x7f7050071850 2024-09-17T05:29:07.764 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:07.764 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:29:07.764 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7030078b90 msgr2=0x7f703007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7030078b90 0x7f703007b0a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7044033000 tx=0x7f7044004730 comp rx=0 tx=0).stop 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 msgr2=0x7f7050158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050158e30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f704000c9a0 tx=0x7f704000ce60 comp rx=0 tx=0).stop 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 shutdown_connections 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7030078b90 0x7f703007b0a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f70500c67a0 0x7f70501f5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70500c4f50 0x7f70500ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 --2- 172.21.15.117:0/1894014237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7050071850 0x7f7050158e30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:07.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 >> 172.21.15.117:0/1894014237 conn(0x7f705006e590 msgr2=0x7f70501483e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:07.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 shutdown_connections 2024-09-17T05:29:07.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:07.765+0000 7f7054ecc640 1 -- 172.21.15.117:0/1894014237 wait complete. 2024-09-17T05:29:07.777 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-09-17T05:29:08.313 INFO:tasks.workunit.client.0.smithi117.stderr:Note: switching to 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'. 2024-09-17T05:29:08.313 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.313 INFO:tasks.workunit.client.0.smithi117.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: git switch -c 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr:Or undo this operation with: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: git switch - 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.314 INFO:tasks.workunit.client.0.smithi117.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T05:29:08.315 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-09-17T05:29:08.315 INFO:tasks.workunit.client.0.smithi117.stderr:HEAD is now at ce1b342bf8c Merge PR #55006 into wip-pdonnell-testing-20240916.200549-debug 2024-09-17T05:29:08.318 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T05:29:08.339 INFO:tasks.workunit.client.0.smithi117.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T05:29:08.341 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T05:29:08.341 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T05:29:08.956 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T05:29:09.022 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T05:29:09.063 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T05:29:09.064 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T05:29:09.064 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T05:29:09.102 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T05:29:09.106 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:29:09.106 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T05:29:09.162 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-09-17T05:29:09.163 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-17T05:29:09.163 DEBUG:teuthology.orchestra.run.smithi117:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce1b342bf8c6349a8c90cd607f83bff2443f21cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-17T05:29:09.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:09.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 -- 172.21.15.117:0/1312674730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69d0005350 msgr2=0x7f69d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:09.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 --2- 172.21.15.117:0/1312674730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69d0005350 0x7f69d000b9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f69e000c260 tx=0x7f69d0005830 comp rx=0 tx=0).stop 2024-09-17T05:29:09.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 -- 172.21.15.117:0/1312674730 shutdown_connections 2024-09-17T05:29:09.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 --2- 172.21.15.117:0/1312674730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69d0005350 0x7f69d000b9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:09.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 -- 172.21.15.117:0/1312674730 >> 172.21.15.117:0/1312674730 conn(0x7f69e806e590 msgr2=0x7f69e80cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:09.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 -- 172.21.15.117:0/1312674730 shutdown_connections 2024-09-17T05:29:09.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.893+0000 7f69ef73a640 1 -- 172.21.15.117:0/1312674730 wait complete. 2024-09-17T05:29:09.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.894+0000 7f69ef73a640 1 Processor -- start 2024-09-17T05:29:09.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.895+0000 7f69ef73a640 1 -- start start 2024-09-17T05:29:09.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ef73a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d0005350 0x7f69e814b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:09.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ef73a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 0x7f69e8155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:09.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ef73a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f69e814dd80 0x7f69e814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ef73a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80c7ff0 con 0x7f69e814a620 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ef73a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80ce0d0 con 0x7f69d0005350 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ef73a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f69e80c83a0 con 0x7f69e814dd80 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 0x7f69e8155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 0x7f69e8155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47884/0 (socket says 172.21.15.117:47884) 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 -- 172.21.15.117:0/2848125328 learned_addr learned my addr 172.21.15.117:0/2848125328 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69ed4d6640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d0005350 0x7f69e814b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69edcd7640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f69e814dd80 0x7f69e814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 -- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f69e814dd80 msgr2=0x7f69e814e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f69e814dd80 0x7f69e814e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 -- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d0005350 msgr2=0x7f69e814b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d0005350 0x7f69e814b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:09.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 -- 172.21.15.117:0/2848125328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e81ef2c0 con 0x7f69e814a620 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69edcd7640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f69e814dd80 0x7f69e814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.896+0000 7f69eccd5640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 0x7f69e8155520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f69dc004e20 tx=0x7f69dc001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.897+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc0107d0 con 0x7f69e814a620 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.897+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69dc016070 con 0x7f69e814a620 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.897+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc010cc0 con 0x7f69e814a620 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.897+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e814d1d0 con 0x7f69e814a620 2024-09-17T05:29:09.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.897+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e81ee7e0 con 0x7f69e814a620 2024-09-17T05:29:09.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.898+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e80c7940 con 0x7f69e814a620 2024-09-17T05:29:09.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.898+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f69dc01b460 con 0x7f69e814a620 2024-09-17T05:29:09.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.899+0000 7f69e5ffb640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f69d4078bc0 0x7f69d407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:09.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.899+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6590+0+0 (secure 0 0 0) 0x7f69dc09b800 con 0x7f69e814a620 2024-09-17T05:29:09.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.899+0000 7f69ed4d6640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f69d4078bc0 0x7f69d407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:09.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.899+0000 7f69ed4d6640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f69d4078bc0 0x7f69d407b0d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f69d8005fd0 tx=0x7f69d8009500 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:09.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:09.903+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f69dc0639d0 con 0x7f69e814a620 2024-09-17T05:29:10.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.115+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69e80c8d00 con 0x7f69e814a620 2024-09-17T05:29:10.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.116+0000 7f69e5ffb640 1 -- 172.21.15.117:0/2848125328 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6922 (secure 0 0 0) 0x7f69dc0632d0 con 0x7f69e814a620 2024-09-17T05:29:10.116 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:10.116 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":14,"btime":"2024-09-17T05:27:55:772732+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4419,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:27:55.772725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4422":{"gid":4422,"name":"e","rank":2,"incarnation":13,"state":"up:active","state_seq":10,"addr":"172.21.15.117:6837/242246295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":242246295},{"type":"v1","addr":"172.21.15.117:6837","nonce":242246295}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4422,4434,4440]},"id":1}]} 2024-09-17T05:29:10.117 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 14 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f69d4078bc0 msgr2=0x7f69d407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f69d4078bc0 0x7f69d407b0d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f69d8005fd0 tx=0x7f69d8009500 comp rx=0 tx=0).stop 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 msgr2=0x7f69e8155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 0x7f69e8155520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f69dc004e20 tx=0x7f69dc001630 comp rx=0 tx=0).stop 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 shutdown_connections 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f69d4078bc0 0x7f69d407b0d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f69e814dd80 0x7f69e814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69e814a620 0x7f69e8155520 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 --2- 172.21.15.117:0/2848125328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d0005350 0x7f69e814b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:10.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 >> 172.21.15.117:0/2848125328 conn(0x7f69e806e590 msgr2=0x7f69e8158130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:10.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 shutdown_connections 2024-09-17T05:29:10.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:10.118+0000 7f69ef73a640 1 -- 172.21.15.117:0/2848125328 wait complete. 2024-09-17T05:29:10.130 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-09-17T05:29:12.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.257+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/1483315721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c3eb0 msgr2=0x7f0d5815bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.257+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/1483315721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c3eb0 0x7f0d5815bdf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0d54005fd0 tx=0x7f0d5402ab20 comp rx=0 tx=0).stop 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.257+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/1483315721 shutdown_connections 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.257+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/1483315721 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c8b70 0x7f0d580c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.257+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/1483315721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c3eb0 0x7f0d5815bdf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.257+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/1483315721 >> 172.21.15.117:0/1483315721 conn(0x7f0d5806cda0 msgr2=0x7f0d5806d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:12.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.258+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/1483315721 shutdown_connections 2024-09-17T05:29:12.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.258+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/1483315721 wait complete. 2024-09-17T05:29:12.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.259+0000 7f0d5ffe4640 1 Processor -- start 2024-09-17T05:29:12.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.259+0000 7f0d5ffe4640 1 -- start start 2024-09-17T05:29:12.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.260+0000 7f0d5ffe4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c3eb0 0x7f0d580c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.260+0000 7f0d5ffe4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d580c7b10 0x7f0d581f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.260+0000 7f0d5ffe4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 0x7f0d58156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.260+0000 7f0d5ffe4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5815b5f0 con 0x7f0d580c8b70 2024-09-17T05:29:12.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.260+0000 7f0d5ffe4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5806e4c0 con 0x7f0d580c7b10 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.260+0000 7f0d5ffe4640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5815b8d0 con 0x7f0d580c3eb0 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 0x7f0d58156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 0x7f0d58156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47908/0 (socket says 172.21.15.117:47908) 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 -- 172.21.15.117:0/988473473 learned_addr learned my addr 172.21.15.117:0/988473473 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5dd80640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c3eb0 0x7f0d580c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5d57f640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d580c7b10 0x7f0d581f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 -- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c3eb0 msgr2=0x7f0d580c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c3eb0 0x7f0d580c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 -- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d580c7b10 msgr2=0x7f0d581f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d580c7b10 0x7f0d581f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 -- 172.21.15.117:0/988473473 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d580c6750 con 0x7f0d580c8b70 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5dd80640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c3eb0 0x7f0d580c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:29:12.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5e581640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 0x7f0d58156b90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c3c0 tx=0x7f0d5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:12.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50017070 con 0x7f0d580c8b70 2024-09-17T05:29:12.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d500053c0 con 0x7f0d580c8b70 2024-09-17T05:29:12.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d580c4cd0 con 0x7f0d580c8b70 2024-09-17T05:29:12.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50003850 con 0x7f0d580c8b70 2024-09-17T05:29:12.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.261+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d581e6e10 con 0x7f0d580c8b70 2024-09-17T05:29:12.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.262+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d580c21a0 con 0x7f0d580c8b70 2024-09-17T05:29:12.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.263+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d5001a070 con 0x7f0d580c8b70 2024-09-17T05:29:12.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.264+0000 7f0d4e7fc640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d3c078bc0 0x7f0d3c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.264+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f0d5009a4b0 con 0x7f0d580c8b70 2024-09-17T05:29:12.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.264+0000 7f0d5dd80640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d3c078bc0 0x7f0d3c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.264+0000 7f0d5dd80640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d3c078bc0 0x7f0d3c07b0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d54035000 tx=0x7f0d54004ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:12.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.268+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d580c21a0 con 0x7f0d580c8b70 2024-09-17T05:29:12.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.475+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d58051080 con 0x7f0d580c8b70 2024-09-17T05:29:12.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.475+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/988473473 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6093 (secure 0 0 0) 0x7f0d58051080 con 0x7f0d580c8b70 2024-09-17T05:29:12.476 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:12.476 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":15,"btime":"2024-09-17T05:29:10:831268+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:10.831263+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:29:12.476 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 15 2024-09-17T05:29:12.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d3c078bc0 msgr2=0x7f0d3c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d3c078bc0 0x7f0d3c07b0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d54035000 tx=0x7f0d54004ab0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 msgr2=0x7f0d58156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 0x7f0d58156b90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c3c0 tx=0x7f0d5000c880 comp rx=0 tx=0).stop 2024-09-17T05:29:12.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 shutdown_connections 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d3c078bc0 0x7f0d3c07b0d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d580c8b70 0x7f0d58156b90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d580c7b10 0x7f0d581f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 --2- 172.21.15.117:0/988473473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d580c3eb0 0x7f0d580c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 >> 172.21.15.117:0/988473473 conn(0x7f0d5806cda0 msgr2=0x7f0d5806ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 shutdown_connections 2024-09-17T05:29:12.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.477+0000 7f0d5ffe4640 1 -- 172.21.15.117:0/988473473 wait complete. 2024-09-17T05:29:12.489 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-09-17T05:29:12.489 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-09-17T05:29:12.489 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:12.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.599+0000 7f5dfedeb640 1 -- 172.21.15.117:0/3892565245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c49e0 msgr2=0x7f5df80c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.599+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/3892565245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c49e0 0x7f5df80c4df0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009c60 tx=0x7f5df002aa50 comp rx=0 tx=0).stop 2024-09-17T05:29:12.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.599+0000 7f5dfedeb640 1 -- 172.21.15.117:0/3892565245 shutdown_connections 2024-09-17T05:29:12.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.599+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/3892565245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c49e0 0x7f5df80c4df0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.600+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/3892565245 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5df80c69b0 0x7f5df8157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.600+0000 7f5dfedeb640 1 -- 172.21.15.117:0/3892565245 >> 172.21.15.117:0/3892565245 conn(0x7f5df80cd5e0 msgr2=0x7f5df80cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:12.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.600+0000 7f5dfedeb640 1 -- 172.21.15.117:0/3892565245 shutdown_connections 2024-09-17T05:29:12.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.600+0000 7f5dfedeb640 1 -- 172.21.15.117:0/3892565245 wait complete. 2024-09-17T05:29:12.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.601+0000 7f5dfedeb640 1 Processor -- start 2024-09-17T05:29:12.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.601+0000 7f5dfedeb640 1 -- start start 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df80c49e0 0x7f5df81e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 0x7f5df81f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5df80c7f60 0x7f5df81519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5df80c93e0 con 0x7f5df80c69b0 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5df80c7740 con 0x7f5df80c49e0 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5df80c9560 con 0x7f5df80c7f60 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 0x7f5df81f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 0x7f5df81f29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47926/0 (socket says 172.21.15.117:47926) 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 -- 172.21.15.117:0/1769904914 learned_addr learned my addr 172.21.15.117:0/1769904914 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfcb87640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df80c49e0 0x7f5df81e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfd388640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5df80c7f60 0x7f5df81519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 -- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5df80c7f60 msgr2=0x7f5df81519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5df80c7f60 0x7f5df81519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 -- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df80c49e0 msgr2=0x7f5df81e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df80c49e0 0x7f5df81e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 -- 172.21.15.117:0/1769904914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df81f2ef0 con 0x7f5df80c69b0 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df7fff640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 0x7f5df81f29b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c430 tx=0x7f5dec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:12.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec017070 con 0x7f5df80c69b0 2024-09-17T05:29:12.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dec0053c0 con 0x7f5df80c69b0 2024-09-17T05:29:12.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df81efd20 con 0x7f5df80c69b0 2024-09-17T05:29:12.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec003850 con 0x7f5df80c69b0 2024-09-17T05:29:12.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.603+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df81e48e0 con 0x7f5df80c69b0 2024-09-17T05:29:12.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.605+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df8157f10 con 0x7f5df80c69b0 2024-09-17T05:29:12.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.605+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5dec01a070 con 0x7f5df80c69b0 2024-09-17T05:29:12.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.606+0000 7f5df57fa640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5de4078bc0 0x7f5de407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:12.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.606+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f5dec09a400 con 0x7f5df80c69b0 2024-09-17T05:29:12.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.606+0000 7f5dfcb87640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5de4078bc0 0x7f5de407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:12.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.606+0000 7f5dfcb87640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5de4078bc0 0x7f5de407b0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5de8005fd0 tx=0x7f5de8005850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:12.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.610+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5dec062500 con 0x7f5df80c69b0 2024-09-17T05:29:12.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.821+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5df81efa80 con 0x7f5df80c69b0 2024-09-17T05:29:12.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.821+0000 7f5df57fa640 1 -- 172.21.15.117:0/1769904914 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6093 (secure 0 0 0) 0x7f5dec061e00 con 0x7f5df80c69b0 2024-09-17T05:29:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:12.822 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":15,"btime":"2024-09-17T05:29:10:831268+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:10.831263+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:29:12.822 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 15 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5de4078bc0 msgr2=0x7f5de407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5de4078bc0 0x7f5de407b0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5de8005fd0 tx=0x7f5de8005850 comp rx=0 tx=0).stop 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 msgr2=0x7f5df81f29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 0x7f5df81f29b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c430 tx=0x7f5dec00c8f0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 shutdown_connections 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5de4078bc0 0x7f5de407b0d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5df80c7f60 0x7f5df81519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5df80c69b0 0x7f5df81f29b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 --2- 172.21.15.117:0/1769904914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df80c49e0 0x7f5df81e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:12.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 >> 172.21.15.117:0/1769904914 conn(0x7f5df80cd5e0 msgr2=0x7f5df8074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:12.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 shutdown_connections 2024-09-17T05:29:12.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:12.824+0000 7f5dfedeb640 1 -- 172.21.15.117:0/1769904914 wait complete. 2024-09-17T05:29:12.836 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:29:10:831268+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4419": { "addr": "172.21.15.117:6835/235108685", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 235108685, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 235108685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4419, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4434": { "addr": "172.21.15.119:6835/2224210538", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2224210538, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2224210538, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4434, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:29:10.831263+0000", "qdb_cluster": [ 4434, 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 4434, "mds_2": 4419 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:29:14.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:14.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/978121535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 msgr2=0x7fd9b00051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/978121535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9b00051f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001dc0 tx=0x7fd9b0005830 comp rx=0 tx=0).stop 2024-09-17T05:29:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9c9870640 1 -- 172.21.15.117:0/978121535 reap_dead start 2024-09-17T05:29:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/978121535 shutdown_connections 2024-09-17T05:29:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/978121535 >> 172.21.15.117:0/978121535 conn(0x7fd9c4075e00 msgr2=0x7fd9c406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/978121535 shutdown_connections 2024-09-17T05:29:14.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.965+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/978121535 wait complete. 2024-09-17T05:29:14.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.967+0000 7fd9cb2d3640 1 Processor -- start 2024-09-17T05:29:14.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.967+0000 7fd9cb2d3640 1 -- start start 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9cb2d3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9c41562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9cb2d3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd9c4156ca0 0x7fd9c40c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9cb2d3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9c4151270 0x7fd9c4151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9cb2d3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c40c5da0 con 0x7fd9b0004d40 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9cb2d3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c415f6b0 con 0x7fd9c4151270 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9cb2d3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c41558b0 con 0x7fd9c4156ca0 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9c41562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9c41562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40924/0 (socket says 172.21.15.117:40924) 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 -- 172.21.15.117:0/2429039869 learned_addr learned my addr 172.21.15.117:0/2429039869 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c9870640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9c4151270 0x7fd9c4151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c886e640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd9c4156ca0 0x7fd9c40c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:14.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 -- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd9c4156ca0 msgr2=0x7fd9c40c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd9c4156ca0 0x7fd9c40c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 -- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9c4151270 msgr2=0x7fd9c4151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9c4151270 0x7fd9c4151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 -- 172.21.15.117:0/2429039869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c40c84b0 con 0x7fd9b0004d40 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c9870640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9c4151270 0x7fd9c4151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9c906f640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9c41562f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0011ae0 tx=0x7fd9c0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.968+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c001e070 con 0x7fd9b0004d40 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.969+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9c000fe70 con 0x7fd9b0004d40 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.969+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c4155c80 con 0x7fd9b0004d40 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.969+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0004290 con 0x7fd9b0004d40 2024-09-17T05:29:14.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.969+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c415a690 con 0x7fd9b0004d40 2024-09-17T05:29:14.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.970+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c40c7a30 con 0x7fd9b0004d40 2024-09-17T05:29:14.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.970+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd9c0021070 con 0x7fd9b0004d40 2024-09-17T05:29:14.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.971+0000 7fd9b9ffb640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd9a8078bc0 0x7fd9a807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:14.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.971+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fd9c009bc00 con 0x7fd9b0004d40 2024-09-17T05:29:14.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.971+0000 7fd9c886e640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd9a8078bc0 0x7fd9a807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:14.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.972+0000 7fd9c886e640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd9a8078bc0 0x7fd9a807b0d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4000c00 tx=0x7fd9b4006720 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:14.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:14.976+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd9c00a2080 con 0x7fd9b0004d40 2024-09-17T05:29:15.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.184+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9c40cb810 con 0x7fd9b0004d40 2024-09-17T05:29:15.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.185+0000 7fd9b9ffb640 1 -- 172.21.15.117:0/2429039869 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6093 (secure 0 0 0) 0x7fd9c00636f0 con 0x7fd9b0004d40 2024-09-17T05:29:15.185 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:15.185 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":15,"btime":"2024-09-17T05:29:10:831268+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:10.831263+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:29:15.186 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 15 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd9a8078bc0 msgr2=0x7fd9a807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd9a8078bc0 0x7fd9a807b0d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4000c00 tx=0x7fd9b4006720 comp rx=0 tx=0).stop 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 msgr2=0x7fd9c41562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9c41562f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0011ae0 tx=0x7fd9c0009e90 comp rx=0 tx=0).stop 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 shutdown_connections 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd9a8078bc0 0x7fd9a807b0d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9c4151270 0x7fd9c4151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd9c4156ca0 0x7fd9c40c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 --2- 172.21.15.117:0/2429039869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b0004d40 0x7fd9c41562f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 >> 172.21.15.117:0/2429039869 conn(0x7fd9c4075e00 msgr2=0x7fd9c40d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 shutdown_connections 2024-09-17T05:29:15.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:15.187+0000 7fd9cb2d3640 1 -- 172.21.15.117:0/2429039869 wait complete. 2024-09-17T05:29:15.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:29:17.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 -- 172.21.15.117:0/3519597082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b56c0 msgr2=0x7f35f8166c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 --2- 172.21.15.117:0/3519597082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b56c0 0x7f35f8166c80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f35f0005fd0 tx=0x7f35f002a710 comp rx=0 tx=0).stop 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 -- 172.21.15.117:0/3519597082 shutdown_connections 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 --2- 172.21.15.117:0/3519597082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b56c0 0x7f35f8166c80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 -- 172.21.15.117:0/3519597082 >> 172.21.15.117:0/3519597082 conn(0x7f35f8074d80 msgr2=0x7f35f80d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 -- 172.21.15.117:0/3519597082 shutdown_connections 2024-09-17T05:29:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.325+0000 7f35fddc3640 1 -- 172.21.15.117:0/3519597082 wait complete. 2024-09-17T05:29:17.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.327+0000 7f35fddc3640 1 Processor -- start 2024-09-17T05:29:17.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.327+0000 7f35fddc3640 1 -- start start 2024-09-17T05:29:17.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.328+0000 7f35fddc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 0x7f35f815d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:17.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.328+0000 7f35fddc3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f80b56c0 0x7f35f81f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:17.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.328+0000 7f35fddc3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35f815bd70 0x7f35f8153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:17.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.328+0000 7f35fddc3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81558b0 con 0x7f35f80b52b0 2024-09-17T05:29:17.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.328+0000 7f35fddc3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f35f80cd0d0 con 0x7f35f80b56c0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.328+0000 7f35fddc3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f35f80cb290 con 0x7f35f815bd70 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 0x7f35f815d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 0x7f35f815d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40944/0 (socket says 172.21.15.117:40944) 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 -- 172.21.15.117:0/1829809678 learned_addr learned my addr 172.21.15.117:0/1829809678 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f6ffd640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f80b56c0 0x7f35f81f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f7fff640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35f815bd70 0x7f35f8153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 -- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35f815bd70 msgr2=0x7f35f8153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35f815bd70 0x7f35f8153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 -- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f80b56c0 msgr2=0x7f35f81f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f80b56c0 0x7f35f81f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 -- 172.21.15.117:0/1829809678 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f80d05c0 con 0x7f35f80b52b0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35f77fe640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 0x7f35f815d2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c760 tx=0x7f35e800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8003b70 con 0x7f35f80b52b0 2024-09-17T05:29:17.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35e8003d30 con 0x7f35f80b52b0 2024-09-17T05:29:17.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f8154700 con 0x7f35f80b52b0 2024-09-17T05:29:17.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e800aab0 con 0x7f35f80b52b0 2024-09-17T05:29:17.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.329+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f80c4870 con 0x7f35f80b52b0 2024-09-17T05:29:17.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.330+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f8159bc0 con 0x7f35f80b52b0 2024-09-17T05:29:17.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.331+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f35e800ac50 con 0x7f35f80b52b0 2024-09-17T05:29:17.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.332+0000 7f35dffff640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f35e4078bc0 0x7f35e407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:17.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.332+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f35e809ade0 con 0x7f35f80b52b0 2024-09-17T05:29:17.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.332+0000 7f35f6ffd640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f35e4078bc0 0x7f35e407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:17.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.333+0000 7f35f6ffd640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f35e4078bc0 0x7f35e407b0d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f35ec000f60 tx=0x7f35ec009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:17.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.336+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f35f8159bc0 con 0x7f35f80b52b0 2024-09-17T05:29:17.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.545+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35f80b30e0 con 0x7f35f80b52b0 2024-09-17T05:29:17.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.545+0000 7f35dffff640 1 -- 172.21.15.117:0/1829809678 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6095 (secure 0 0 0) 0x7f35f80b30e0 con 0x7f35f80b52b0 2024-09-17T05:29:17.546 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:17.546 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":16,"btime":"2024-09-17T05:29:15:943257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:15.840232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:resolve","state_seq":30,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:29:17.546 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 16 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f35e4078bc0 msgr2=0x7f35e407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f35e4078bc0 0x7f35e407b0d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f35ec000f60 tx=0x7f35ec009040 comp rx=0 tx=0).stop 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 msgr2=0x7f35f815d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 0x7f35f815d2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c760 tx=0x7f35e800cc20 comp rx=0 tx=0).stop 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 shutdown_connections 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f35e4078bc0 0x7f35e407b0d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35f815bd70 0x7f35f8153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f80b56c0 0x7f35f81f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 --2- 172.21.15.117:0/1829809678 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35f80b52b0 0x7f35f815d2f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:17.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.547+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 >> 172.21.15.117:0/1829809678 conn(0x7f35f8074d80 msgr2=0x7f35f8155000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:17.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.548+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 shutdown_connections 2024-09-17T05:29:17.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:17.548+0000 7f35fddc3640 1 -- 172.21.15.117:0/1829809678 wait complete. 2024-09-17T05:29:17.559 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:29:19.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 -- 172.21.15.117:0/3539370761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 msgr2=0x7fc79c159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 --2- 172.21.15.117:0/3539370761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c159d30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc79c074a20 tx=0x7fc79002a680 comp rx=0 tx=0).stop 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 -- 172.21.15.117:0/3539370761 shutdown_connections 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 --2- 172.21.15.117:0/3539370761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c159d30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 --2- 172.21.15.117:0/3539370761 >> v2:172.21.15.119:3300/0 conn(0x7fc79c0c9680 0x7fc79c1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 -- 172.21.15.117:0/3539370761 >> 172.21.15.117:0/3539370761 conn(0x7fc79c06d790 msgr2=0x7fc79c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 -- 172.21.15.117:0/3539370761 shutdown_connections 2024-09-17T05:29:19.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.685+0000 7fc7a133a640 1 -- 172.21.15.117:0/3539370761 wait complete. 2024-09-17T05:29:19.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.687+0000 7fc7a133a640 1 Processor -- start 2024-09-17T05:29:19.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.687+0000 7fc7a133a640 1 -- start start 2024-09-17T05:29:19.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc7a133a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c15a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:19.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc7a133a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc79c0c9680 0x7fc79c14d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:19.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc7a133a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc79c0ce5d0 0x7fc79c143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:19.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc7a133a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c0ca950 con 0x7fc79c0c9020 2024-09-17T05:29:19.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc7a133a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c0c4a60 con 0x7fc79c0c9680 2024-09-17T05:29:19.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc7a133a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc79c14e140 con 0x7fc79c0ce5d0 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc79affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c15a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc79affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c15a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40972/0 (socket says 172.21.15.117:40972) 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc79affd640 1 -- 172.21.15.117:0/4174381134 learned_addr learned my addr 172.21.15.117:0/4174381134 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc79a7fc640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc79c0c9680 0x7fc79c14d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.688+0000 7fc79b7fe640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc79c0ce5d0 0x7fc79c143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79affd640 1 -- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc79c0ce5d0 msgr2=0x7fc79c143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79affd640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc79c0ce5d0 0x7fc79c143e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79affd640 1 -- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc79c0c9680 msgr2=0x7fc79c14d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79affd640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc79c0c9680 0x7fc79c14d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79affd640 1 -- 172.21.15.117:0/4174381134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c14dc70 con 0x7fc79c0c9020 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79a7fc640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc79c0c9680 0x7fc79c14d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc79affd640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c15a460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d770 tx=0x7fc78c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:19.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c017070 con 0x7fc79c0c9020 2024-09-17T05:29:19.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc78c009cc0 con 0x7fc79c0c9020 2024-09-17T05:29:19.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c1f3aa0 con 0x7fc79c0c9020 2024-09-17T05:29:19.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0106c0 con 0x7fc79c0c9020 2024-09-17T05:29:19.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.689+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c15b080 con 0x7fc79c0c9020 2024-09-17T05:29:19.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.690+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79c153450 con 0x7fc79c0c9020 2024-09-17T05:29:19.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.691+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc78c01a070 con 0x7fc79c0c9020 2024-09-17T05:29:19.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.691+0000 7fc7837fe640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc788078bc0 0x7fc78807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:19.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.691+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fc78c09b1c0 con 0x7fc79c0c9020 2024-09-17T05:29:19.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.691+0000 7fc79a7fc640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc788078bc0 0x7fc78807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:19.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.692+0000 7fc79a7fc640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc788078bc0 0x7fc78807b0d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc790033000 tx=0x7fc790004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:19.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.696+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc78c063240 con 0x7fc79c0c9020 2024-09-17T05:29:19.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.904+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc79c0c3e30 con 0x7fc79c0c9020 2024-09-17T05:29:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.905+0000 7fc7837fe640 1 -- 172.21.15.117:0/4174381134 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6095 (secure 0 0 0) 0x7fc78c062b40 con 0x7fc79c0c9020 2024-09-17T05:29:19.905 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:19.905 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":16,"btime":"2024-09-17T05:29:15:943257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:15.840232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:resolve","state_seq":30,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:29:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 16 2024-09-17T05:29:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc788078bc0 msgr2=0x7fc78807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc788078bc0 0x7fc78807b0d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc790033000 tx=0x7fc790004730 comp rx=0 tx=0).stop 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 msgr2=0x7fc79c15a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c15a460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d770 tx=0x7fc78c00dc30 comp rx=0 tx=0).stop 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 shutdown_connections 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc788078bc0 0x7fc78807b0d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc79c0ce5d0 0x7fc79c143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc79c0c9680 0x7fc79c14d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 --2- 172.21.15.117:0/4174381134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79c0c9020 0x7fc79c15a460 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 >> 172.21.15.117:0/4174381134 conn(0x7fc79c06d790 msgr2=0x7fc79c14fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 shutdown_connections 2024-09-17T05:29:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:19.907+0000 7fc7a133a640 1 -- 172.21.15.117:0/4174381134 wait complete. 2024-09-17T05:29:19.919 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:29:21.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 -- 172.21.15.117:0/1620951943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 msgr2=0x7fb8d80c35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 --2- 172.21.15.117:0/1620951943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c35d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8072760 tx=0x7fb8c802a5f0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 -- 172.21.15.117:0/1620951943 shutdown_connections 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 --2- 172.21.15.117:0/1620951943 >> v2:172.21.15.119:3300/0 conn(0x7fb8d80c3b10 0x7fb8d80c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 --2- 172.21.15.117:0/1620951943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c35d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 -- 172.21.15.117:0/1620951943 >> 172.21.15.117:0/1620951943 conn(0x7fb8d8072cc0 msgr2=0x7fb8d80730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:22.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 -- 172.21.15.117:0/1620951943 shutdown_connections 2024-09-17T05:29:22.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.048+0000 7fb8df939640 1 -- 172.21.15.117:0/1620951943 wait complete. 2024-09-17T05:29:22.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.050+0000 7fb8df939640 1 Processor -- start 2024-09-17T05:29:22.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.050+0000 7fb8df939640 1 -- start start 2024-09-17T05:29:22.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8df939640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8d80c3b10 0x7fb8d80c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:22.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8df939640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb8d8153210 0x7fb8d80c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:22.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8df939640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:22.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8df939640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d80cb790 con 0x7fb8d8159a90 2024-09-17T05:29:22.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8df939640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d815d0d0 con 0x7fb8d80c3b10 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8df939640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb8d8159ea0 con 0x7fb8d8153210 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40988/0 (socket says 172.21.15.117:40988) 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 -- 172.21.15.117:0/1807153860 learned_addr learned my addr 172.21.15.117:0/1807153860 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dd6d5640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8d80c3b10 0x7fb8d80c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dced4640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb8d8153210 0x7fb8d80c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 -- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb8d8153210 msgr2=0x7fb8d80c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb8d8153210 0x7fb8d80c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 -- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8d80c3b10 msgr2=0x7fb8d80c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:22.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.051+0000 7fb8dded6640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8d80c3b10 0x7fb8d80c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8dded6640 1 -- 172.21.15.117:0/1807153860 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d815af20 con 0x7fb8d8159a90 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8dded6640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c79b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000ca80 tx=0x7fb8d000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0007df0 con 0x7fb8d8159a90 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8d0003aa0 con 0x7fb8d8159a90 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8d815c180 con 0x7fb8d8159a90 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0004030 con 0x7fb8d8159a90 2024-09-17T05:29:22.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.052+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8d81eff70 con 0x7fb8d8159a90 2024-09-17T05:29:22.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.053+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8d81f0350 con 0x7fb8d8159a90 2024-09-17T05:29:22.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.054+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb8d00041d0 con 0x7fb8d8159a90 2024-09-17T05:29:22.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.054+0000 7fb8cdffb640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb8bc078bc0 0x7fb8bc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:22.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.055+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fb8d009bb20 con 0x7fb8d8159a90 2024-09-17T05:29:22.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.055+0000 7fb8dd6d5640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb8bc078bc0 0x7fb8bc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:22.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.055+0000 7fb8dd6d5640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb8bc078bc0 0x7fb8bc07b0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8d80cb020 tx=0x7fb8d4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.059+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb8d81f0350 con 0x7fb8d8159a90 2024-09-17T05:29:22.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.270+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8d80c35d0 con 0x7fb8d8159a90 2024-09-17T05:29:22.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.271+0000 7fb8cdffb640 1 -- 172.21.15.117:0/1807153860 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6097 (secure 0 0 0) 0x7fb8d0014070 con 0x7fb8d8159a90 2024-09-17T05:29:22.271 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:22.271 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":17,"btime":"2024-09-17T05:29:20:964178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:20.949350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:29:22.272 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 17 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb8bc078bc0 msgr2=0x7fb8bc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb8bc078bc0 0x7fb8bc07b0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8d80cb020 tx=0x7fb8d4009040 comp rx=0 tx=0).stop 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 msgr2=0x7fb8d80c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c79b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000ca80 tx=0x7fb8d000cf40 comp rx=0 tx=0).stop 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 shutdown_connections 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb8bc078bc0 0x7fb8bc07b0d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8d8159a90 0x7fb8d80c79b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb8d8153210 0x7fb8d80c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 --2- 172.21.15.117:0/1807153860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8d80c3b10 0x7fb8d80c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 >> 172.21.15.117:0/1807153860 conn(0x7fb8d8072cc0 msgr2=0x7fb8d81492e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 shutdown_connections 2024-09-17T05:29:22.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:22.273+0000 7fb8df939640 1 -- 172.21.15.117:0/1807153860 wait complete. 2024-09-17T05:29:22.285 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:29:22.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:29:22.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:29:24.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:24.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21f939640 1 -- 172.21.15.117:0/2875457193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe204004000 msgr2=0x7fe204004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:24.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21f939640 1 --2- 172.21.15.117:0/2875457193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe204004000 0x7fe204004800 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe208001690 tx=0x7fe204005830 comp rx=0 tx=0).stop 2024-09-17T05:29:24.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21ded6640 1 -- 172.21.15.117:0/2875457193 reap_dead start 2024-09-17T05:29:24.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21f939640 1 -- 172.21.15.117:0/2875457193 shutdown_connections 2024-09-17T05:29:24.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21f939640 1 -- 172.21.15.117:0/2875457193 >> 172.21.15.117:0/2875457193 conn(0x7fe218074d80 msgr2=0x7fe2180d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:24.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21f939640 1 -- 172.21.15.117:0/2875457193 shutdown_connections 2024-09-17T05:29:24.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.411+0000 7fe21f939640 1 -- 172.21.15.117:0/2875457193 wait complete. 2024-09-17T05:29:24.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.413+0000 7fe21f939640 1 Processor -- start 2024-09-17T05:29:24.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.413+0000 7fe21f939640 1 -- start start 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21f939640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe204004000 0x7fe21815e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21f939640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 0x7fe2181f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21f939640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe218157f90 0x7fe218152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21f939640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181558b0 con 0x7fe2181f1d50 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21f939640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe2180cd0d0 con 0x7fe204004000 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21f939640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe2180cb290 con 0x7fe218157f90 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21ced4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 0x7fe2181f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21ced4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 0x7fe2181f4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38196/0 (socket says 172.21.15.117:38196) 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21ced4640 1 -- 172.21.15.117:0/2665368463 learned_addr learned my addr 172.21.15.117:0/2665368463 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21ded6640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe218157f90 0x7fe218152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.414+0000 7fe21d6d5640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe204004000 0x7fe21815e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:24.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21ced4640 1 -- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe218157f90 msgr2=0x7fe218152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21ced4640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe218157f90 0x7fe218152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21ced4640 1 -- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe204004000 msgr2=0x7fe21815e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21ced4640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe204004000 0x7fe21815e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21ced4640 1 -- 172.21.15.117:0/2665368463 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2181f5180 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21ced4640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 0x7fe2181f4c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe20802a910 tx=0x7fe208006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208038070 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe208003af0 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe21815eaa0 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20803d460 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.415+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21815f060 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.416+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe21806af90 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.417+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe208030070 con 0x7fe2181f1d50 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.417+0000 7fe20dffb640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1fc078bc0 0x7fe1fc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:24.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.418+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fe2080bacb0 con 0x7fe2181f1d50 2024-09-17T05:29:24.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.418+0000 7fe21d6d5640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1fc078bc0 0x7fe1fc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:24.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.418+0000 7fe21d6d5640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1fc078bc0 0x7fe1fc07b0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe214009c50 tx=0x7fe214005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:24.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.422+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe21806af90 con 0x7fe2181f1d50 2024-09-17T05:29:24.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.632+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2181f4860 con 0x7fe2181f1d50 2024-09-17T05:29:24.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.632+0000 7fe20dffb640 1 -- 172.21.15.117:0/2665368463 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6097 (secure 0 0 0) 0x7fe208042060 con 0x7fe2181f1d50 2024-09-17T05:29:24.633 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:24.633 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":17,"btime":"2024-09-17T05:29:20:964178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:20.949350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:29:24.633 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 17 2024-09-17T05:29:24.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1fc078bc0 msgr2=0x7fe1fc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:24.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1fc078bc0 0x7fe1fc07b0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe214009c50 tx=0x7fe214005f90 comp rx=0 tx=0).stop 2024-09-17T05:29:24.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 msgr2=0x7fe2181f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 0x7fe2181f4c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe20802a910 tx=0x7fe208006360 comp rx=0 tx=0).stop 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 shutdown_connections 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1fc078bc0 0x7fe1fc07b0d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe218157f90 0x7fe218152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2181f1d50 0x7fe2181f4c40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 --2- 172.21.15.117:0/2665368463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe204004000 0x7fe21815e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 >> 172.21.15.117:0/2665368463 conn(0x7fe218074d80 msgr2=0x7fe218154de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 shutdown_connections 2024-09-17T05:29:24.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:24.635+0000 7fe21f939640 1 -- 172.21.15.117:0/2665368463 wait complete. 2024-09-17T05:29:24.648 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:29:20:964178+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4419": { "addr": "172.21.15.117:6835/235108685", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 235108685, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 235108685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4419, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 32 }, "gid_4434": { "addr": "172.21.15.119:6835/2224210538", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2224210538, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2224210538, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4434, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:29:20.949350+0000", "qdb_cluster": [ 4434, 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 4434, "mds_2": 4419 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:29:26.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 -- 172.21.15.117:0/3386030818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c0006dbe0 msgr2=0x7f6c0006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 --2- 172.21.15.117:0/3386030818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c0006dbe0 0x7f6c0006dff0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6c000d14f0 tx=0x7f6bf802aac0 comp rx=0 tx=0).stop 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 -- 172.21.15.117:0/3386030818 shutdown_connections 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 --2- 172.21.15.117:0/3386030818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c0006dbe0 0x7f6c0006dff0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 --2- 172.21.15.117:0/3386030818 >> v2:172.21.15.119:3300/0 conn(0x7f6c000b52b0 0x7f6c000b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 --2- 172.21.15.117:0/3386030818 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6c0015bd70 0x7f6c000ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 -- 172.21.15.117:0/3386030818 >> 172.21.15.117:0/3386030818 conn(0x7f6c00074d80 msgr2=0x7f6c000d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:26.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 -- 172.21.15.117:0/3386030818 shutdown_connections 2024-09-17T05:29:26.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.776+0000 7f6c06950640 1 -- 172.21.15.117:0/3386030818 wait complete. 2024-09-17T05:29:26.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.778+0000 7f6c06950640 1 Processor -- start 2024-09-17T05:29:26.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.778+0000 7f6c06950640 1 -- start start 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c06950640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6c0006dbe0 0x7f6c00157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c06950640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 0x7f6c001f1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c06950640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0015bd70 0x7f6c00154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c06950640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6c000cacb0 con 0x7f6c000b52b0 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c06950640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6c000cd0d0 con 0x7f6c0015bd70 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c06950640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6c001558b0 con 0x7f6c0006dbe0 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6bff7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 0x7f6c001f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:26.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6bff7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 0x7f6c001f1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38226/0 (socket says 172.21.15.117:38226) 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6bff7fe640 1 -- 172.21.15.117:0/2378374143 learned_addr learned my addr 172.21.15.117:0/2378374143 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6bfffff640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6c0006dbe0 0x7f6c00157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6c04eed640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0015bd70 0x7f6c00154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6bff7fe640 1 -- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6c0006dbe0 msgr2=0x7f6c00157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.779+0000 7f6bff7fe640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6c0006dbe0 0x7f6c00157620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bff7fe640 1 -- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0015bd70 msgr2=0x7f6c00154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bff7fe640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0015bd70 0x7f6c00154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bff7fe640 1 -- 172.21.15.117:0/2378374143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0015d720 con 0x7f6c000b52b0 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bff7fe640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 0x7f6c001f1410 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400d6c0 tx=0x7f6bf400db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4017070 con 0x7f6c000b52b0 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bf4009ab0 con 0x7f6c000b52b0 2024-09-17T05:29:26.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0015d9c0 con 0x7f6c000b52b0 2024-09-17T05:29:26.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4010460 con 0x7f6c000b52b0 2024-09-17T05:29:26.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.780+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c00155c80 con 0x7f6c000b52b0 2024-09-17T05:29:26.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.781+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c000cb290 con 0x7f6c000b52b0 2024-09-17T05:29:26.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.782+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6bf401a070 con 0x7f6c000b52b0 2024-09-17T05:29:26.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.782+0000 7f6bfcff9640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bec078bc0 0x7f6bec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:26.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.782+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f6bf409aff0 con 0x7f6c000b52b0 2024-09-17T05:29:26.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.782+0000 7f6bfffff640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bec078bc0 0x7f6bec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:26.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.783+0000 7f6bfffff640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bec078bc0 0x7f6bec07b0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0004630 tx=0x7f6bf000b2e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:26.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.787+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6bf4062a10 con 0x7f6c000b52b0 2024-09-17T05:29:26.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.998+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c000cc030 con 0x7f6c000b52b0 2024-09-17T05:29:26.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:26.998+0000 7f6bfcff9640 1 -- 172.21.15.117:0/2378374143 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6094 (secure 0 0 0) 0x7f6bf4067440 con 0x7f6c000b52b0 2024-09-17T05:29:26.999 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:26.999 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":18,"btime":"2024-09-17T05:29:25:983404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:25.966553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:29:26.999 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 18 2024-09-17T05:29:27.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bec078bc0 msgr2=0x7f6bec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:27.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bec078bc0 0x7f6bec07b0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0004630 tx=0x7f6bf000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:29:27.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 msgr2=0x7f6c001f1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 0x7f6c001f1410 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400d6c0 tx=0x7f6bf400db80 comp rx=0 tx=0).stop 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 shutdown_connections 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6bec078bc0 0x7f6bec07b0d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0015bd70 0x7f6c00154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c000b52b0 0x7f6c001f1410 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 --2- 172.21.15.117:0/2378374143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6c0006dbe0 0x7f6c00157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 >> 172.21.15.117:0/2378374143 conn(0x7f6c00074d80 msgr2=0x7f6c000cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 shutdown_connections 2024-09-17T05:29:27.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:27.001+0000 7f6c06950640 1 -- 172.21.15.117:0/2378374143 wait complete. 2024-09-17T05:29:27.013 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:29:29.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3022988532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce415bd70 msgr2=0x7f6ce40ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3022988532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce415bd70 0x7f6ce40ca770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4158e50 tx=0x7f6cd402a680 comp rx=0 tx=0).stop 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3022988532 shutdown_connections 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3022988532 >> v2:172.21.15.119:3300/0 conn(0x7f6ce40b52b0 0x7f6ce40b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3022988532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce415bd70 0x7f6ce40ca770 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3022988532 >> 172.21.15.117:0/3022988532 conn(0x7f6ce4074d80 msgr2=0x7f6ce40d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3022988532 shutdown_connections 2024-09-17T05:29:29.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.139+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3022988532 wait complete. 2024-09-17T05:29:29.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.141+0000 7f6ceb21f640 1 Processor -- start 2024-09-17T05:29:29.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.141+0000 7f6ceb21f640 1 -- start start 2024-09-17T05:29:29.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ceb21f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 0x7f6ce41f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ceb21f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ce40b52b0 0x7f6ce40d0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ceb21f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce415bd70 0x7f6ce415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ceb21f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce40cacb0 con 0x7f6ce406dbe0 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ceb21f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce40cd0d0 con 0x7f6ce40b52b0 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ceb21f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce41558b0 con 0x7f6ce415bd70 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ce8fbb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 0x7f6ce41f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ce8fbb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 0x7f6ce41f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38234/0 (socket says 172.21.15.117:38234) 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ce8fbb640 1 -- 172.21.15.117:0/3107268279 learned_addr learned my addr 172.21.15.117:0/3107268279 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.142+0000 7f6ce3fff640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ce40b52b0 0x7f6ce40d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce97bc640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce415bd70 0x7f6ce415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce8fbb640 1 -- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce415bd70 msgr2=0x7f6ce415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:29.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce8fbb640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce415bd70 0x7f6ce415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce8fbb640 1 -- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ce40b52b0 msgr2=0x7f6ce40d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce8fbb640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ce40b52b0 0x7f6ce40d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce8fbb640 1 -- 172.21.15.117:0/3107268279 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce41f2790 con 0x7f6ce406dbe0 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce97bc640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce415bd70 0x7f6ce415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce3fff640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ce40b52b0 0x7f6ce40d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce8fbb640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 0x7f6ce41f2250 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4034040 tx=0x7f6cd4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd403e030 con 0x7f6ce406dbe0 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cd4003d10 con 0x7f6ce406dbe0 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce40c83d0 con 0x7f6ce406dbe0 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4035040 con 0x7f6ce406dbe0 2024-09-17T05:29:29.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.143+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce40d0d40 con 0x7f6ce406dbe0 2024-09-17T05:29:29.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.144+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce40cc410 con 0x7f6ce406dbe0 2024-09-17T05:29:29.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.145+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6cd4041030 con 0x7f6ce406dbe0 2024-09-17T05:29:29.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.145+0000 7f6ce17fa640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cd0078b90 0x7f6cd007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:29.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.146+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f6cd40bb950 con 0x7f6ce406dbe0 2024-09-17T05:29:29.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.146+0000 7f6ce3fff640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cd0078b90 0x7f6cd007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:29.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.146+0000 7f6ce3fff640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cd0078b90 0x7f6cd007b0a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ce415dba0 tx=0x7f6cd8009500 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:29.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.150+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6cd4083940 con 0x7f6ce406dbe0 2024-09-17T05:29:29.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.356+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ce40cecb0 con 0x7f6ce406dbe0 2024-09-17T05:29:29.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.357+0000 7f6ce17fa640 1 -- 172.21.15.117:0/3107268279 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6094 (secure 0 0 0) 0x7f6cd4083240 con 0x7f6ce406dbe0 2024-09-17T05:29:29.358 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:29.358 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":18,"btime":"2024-09-17T05:29:25:983404+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:25.966553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:29:29.359 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 18 2024-09-17T05:29:29.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cd0078b90 msgr2=0x7f6cd007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cd0078b90 0x7f6cd007b0a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ce415dba0 tx=0x7f6cd8009500 comp rx=0 tx=0).stop 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 msgr2=0x7f6ce41f2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 0x7f6ce41f2250 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4034040 tx=0x7f6cd4003820 comp rx=0 tx=0).stop 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 shutdown_connections 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6cd0078b90 0x7f6cd007b0a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ce415bd70 0x7f6ce415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ce40b52b0 0x7f6ce40d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 --2- 172.21.15.117:0/3107268279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce406dbe0 0x7f6ce41f2250 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 >> 172.21.15.117:0/3107268279 conn(0x7f6ce4074d80 msgr2=0x7f6ce4154f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 shutdown_connections 2024-09-17T05:29:29.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:29.359+0000 7f6ceb21f640 1 -- 172.21.15.117:0/3107268279 wait complete. 2024-09-17T05:29:29.372 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:29:30.996 INFO:tasks.workunit.client.0.smithi117.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-17T05:29:30.997 INFO:tasks.workunit.client.0.smithi117.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-09-17T05:29:30.997 INFO:tasks.workunit.client.0.smithi117.stderr:+ trap -- retry EXIT 2024-09-17T05:29:30.997 INFO:tasks.workunit.client.0.smithi117.stderr:+ rm -rf ffsb 2024-09-17T05:29:30.997 INFO:tasks.workunit.client.0.smithi117.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-17T05:29:31.001 INFO:tasks.workunit.client.0.smithi117.stderr:Cloning into 'ffsb'... 2024-09-17T05:29:31.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:31.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 -- 172.21.15.117:0/1464224299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 msgr2=0x7f9090157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 --2- 172.21.15.117:0/1464224299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f9090157f10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f90900732f0 tx=0x7f908c02a5f0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 -- 172.21.15.117:0/1464224299 shutdown_connections 2024-09-17T05:29:31.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 --2- 172.21.15.117:0/1464224299 >> v2:172.21.15.119:3300/0 conn(0x7f90900c7f60 0x7f9090141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 --2- 172.21.15.117:0/1464224299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f9090157f10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 -- 172.21.15.117:0/1464224299 >> 172.21.15.117:0/1464224299 conn(0x7f90900cd5e0 msgr2=0x7f90900cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 -- 172.21.15.117:0/1464224299 shutdown_connections 2024-09-17T05:29:31.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.492+0000 7f9097bdf640 1 -- 172.21.15.117:0/1464224299 wait complete. 2024-09-17T05:29:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.494+0000 7f9097bdf640 1 Processor -- start 2024-09-17T05:29:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.494+0000 7f9097bdf640 1 -- start start 2024-09-17T05:29:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.495+0000 7f9097bdf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90900c49e0 0x7f90901f10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.495+0000 7f9097bdf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f909013c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.495+0000 7f9097bdf640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90900c7f60 0x7f90900cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.495+0000 7f9097bdf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9090141690 con 0x7f90900c69b0 2024-09-17T05:29:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.495+0000 7f9097bdf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f90900c7740 con 0x7f90900c49e0 2024-09-17T05:29:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.495+0000 7f9097bdf640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f90900c9af0 con 0x7f90900c7f60 2024-09-17T05:29:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f909013c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f909013c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38242/0 (socket says 172.21.15.117:38242) 2024-09-17T05:29:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 -- 172.21.15.117:0/3555456492 learned_addr learned my addr 172.21.15.117:0/3555456492 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909617c640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90900c7f60 0x7f90900cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909597b640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90900c49e0 0x7f90901f10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 -- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90900c7f60 msgr2=0x7f90900cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90900c7f60 0x7f90900cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 -- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90900c49e0 msgr2=0x7f90901f10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90900c49e0 0x7f90901f10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 -- 172.21.15.117:0/3555456492 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90901f1600 con 0x7f90900c69b0 2024-09-17T05:29:31.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f909517a640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f909013c0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f908400e790 tx=0x7f908400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:31.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908400ca60 con 0x7f90900c69b0 2024-09-17T05:29:31.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f908400ccf0 con 0x7f90900c69b0 2024-09-17T05:29:31.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909013fbf0 con 0x7f90900c69b0 2024-09-17T05:29:31.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084010690 con 0x7f90900c69b0 2024-09-17T05:29:31.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.496+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9090152c10 con 0x7f90900c69b0 2024-09-17T05:29:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.498+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9090141150 con 0x7f90900c69b0 2024-09-17T05:29:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.498+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9084010830 con 0x7f90900c69b0 2024-09-17T05:29:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.499+0000 7f90827fc640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9074078b90 0x7f907407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.499+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f908409ba40 con 0x7f90900c69b0 2024-09-17T05:29:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.499+0000 7f909597b640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9074078b90 0x7f907407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.500+0000 7f909597b640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9074078b90 0x7f907407b0a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f908c033000 tx=0x7f908c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:31.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.504+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f90840634f0 con 0x7f90900c69b0 2024-09-17T05:29:31.634 INFO:tasks.ceph.mds.f.smithi119.stderr:2024-09-17T05:29:31.633+0000 7f6f66926640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-17T05:29:31.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.725+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90900742c0 con 0x7f90900c69b0 2024-09-17T05:29:31.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.726+0000 7f90827fc640 1 -- 172.21.15.117:0/3555456492 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6099 (secure 0 0 0) 0x7f9084067f20 con 0x7f90900c69b0 2024-09-17T05:29:31.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:31.728 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":19,"btime":"2024-09-17T05:29:31:001989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4419,4434,4440]},"id":1}]} 2024-09-17T05:29:31.730 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 19 2024-09-17T05:29:31.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9074078b90 msgr2=0x7f907407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9074078b90 0x7f907407b0a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f908c033000 tx=0x7f908c004730 comp rx=0 tx=0).stop 2024-09-17T05:29:31.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 msgr2=0x7f909013c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f909013c0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f908400e790 tx=0x7f908400ec50 comp rx=0 tx=0).stop 2024-09-17T05:29:31.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 shutdown_connections 2024-09-17T05:29:31.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9074078b90 0x7f907407b0a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90900c7f60 0x7f90900cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90900c69b0 0x7f909013c0e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 --2- 172.21.15.117:0/3555456492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90900c49e0 0x7f90901f10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 >> 172.21.15.117:0/3555456492 conn(0x7f90900cd5e0 msgr2=0x7f9090140140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:31.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 shutdown_connections 2024-09-17T05:29:31.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.728+0000 7f9097bdf640 1 -- 172.21.15.117:0/3555456492 wait complete. 2024-09-17T05:29:31.741 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4422 2024-09-17T05:29:31.742 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.e 2024-09-17T05:29:31.742 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-09-17T05:29:31.742 INFO:tasks.ceph.mds.e:Restarting daemon 2024-09-17T05:29:31.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-09-17T05:29:31.744 INFO:tasks.ceph.mds.e:Started 2024-09-17T05:29:31.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:29:31.832 INFO:tasks.ceph.mds.e.smithi117.stdout:starting mds.e at 2024-09-17T05:29:31.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.858+0000 7f5205cf2640 1 -- 172.21.15.117:0/1995050652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5200152210 msgr2=0x7f52000c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.858+0000 7f5205cf2640 1 --2- 172.21.15.117:0/1995050652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5200152210 0x7f52000c4f50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009830 tx=0x7f51e802a480 comp rx=0 tx=0).stop 2024-09-17T05:29:31.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.858+0000 7f5205cf2640 1 -- 172.21.15.117:0/1995050652 shutdown_connections 2024-09-17T05:29:31.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.858+0000 7f5205cf2640 1 --2- 172.21.15.117:0/1995050652 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c5490 0x7f5200162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.858+0000 7f5205cf2640 1 --2- 172.21.15.117:0/1995050652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5200152210 0x7f52000c4f50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.858+0000 7f5205cf2640 1 -- 172.21.15.117:0/1995050652 >> 172.21.15.117:0/1995050652 conn(0x7f520006e590 msgr2=0x7f52000cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:31.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.859+0000 7f5205cf2640 1 -- 172.21.15.117:0/1995050652 shutdown_connections 2024-09-17T05:29:31.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.859+0000 7f5205cf2640 1 -- 172.21.15.117:0/1995050652 wait complete. 2024-09-17T05:29:31.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.860+0000 7f5205cf2640 1 Processor -- start 2024-09-17T05:29:31.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.860+0000 7f5205cf2640 1 -- start start 2024-09-17T05:29:31.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f5205cf2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 0x7f52001f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f5205cf2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c67a0 0x7f52001411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f5205cf2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5200152210 0x7f52000ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f5205cf2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f52000ce0d0 con 0x7f52000c5490 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f5205cf2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f52000c58a0 con 0x7f5200152210 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f5205cf2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f52000c7ff0 con 0x7f52000c67a0 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51ff7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 0x7f52001f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c67a0 0x7f52001411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51ff7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 0x7f52001f0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38286/0 (socket says 172.21.15.117:38286) 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51ff7fe640 1 -- 172.21.15.117:0/3077462499 learned_addr learned my addr 172.21.15.117:0/3077462499 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51fffff640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5200152210 0x7f52000ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 -- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5200152210 msgr2=0x7f52000ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5200152210 0x7f52000ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 -- 172.21.15.117:0/3077462499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 msgr2=0x7f52001f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 0x7f52001f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:31.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 -- 172.21.15.117:0/3077462499 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52001f08a0 con 0x7f52000c67a0 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51ff7fe640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 0x7f52001f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51feffd640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c67a0 0x7f52001411e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51e8031000 tx=0x7f51e8033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8031b50 con 0x7f52000c67a0 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.862+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51e8031de0 con 0x7f52000c67a0 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.863+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8005280 con 0x7f52000c67a0 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.863+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52001513d0 con 0x7f52000c67a0 2024-09-17T05:29:31.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.863+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5200151990 con 0x7f52000c67a0 2024-09-17T05:29:31.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.863+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52000c89b0 con 0x7f52000c67a0 2024-09-17T05:29:31.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.864+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f51e8005420 con 0x7f52000c67a0 2024-09-17T05:29:31.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.865+0000 7f51f7fff640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f51e0078b90 0x7f51e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:29:31.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.865+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f51e80bc4e0 con 0x7f52000c67a0 2024-09-17T05:29:31.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.865+0000 7f51ff7fe640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f51e0078b90 0x7f51e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:29:31.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.865+0000 7f51ff7fe640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f51e0078b90 0x7f51e007b0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f51f0001bd0 tx=0x7f51f0007f50 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:29:31.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:31.870+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f51e803f030 con 0x7f52000c67a0 2024-09-17T05:29:31.931 INFO:tasks.workunit.client.0.smithi117.stderr:+ trap - EXIT 2024-09-17T05:29:31.931 INFO:tasks.workunit.client.0.smithi117.stderr:+ cd ffsb 2024-09-17T05:29:31.932 INFO:tasks.workunit.client.0.smithi117.stderr:+ ./configure 2024-09-17T05:29:32.048 INFO:tasks.workunit.client.0.smithi117.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-17T05:29:32.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.092+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52000c8d00 con 0x7f52000c67a0 2024-09-17T05:29:32.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.093+0000 7f51f7fff640 1 -- 172.21.15.117:0/3077462499 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7f51e8031770 con 0x7f52000c67a0 2024-09-17T05:29:32.095 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:29:32.095 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:29:32.095 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:29:32.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f51e0078b90 msgr2=0x7f51e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f51e0078b90 0x7f51e007b0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f51f0001bd0 tx=0x7f51f0007f50 comp rx=0 tx=0).stop 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c67a0 msgr2=0x7f52001411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c67a0 0x7f52001411e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51e8031000 tx=0x7f51e8033ba0 comp rx=0 tx=0).stop 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 shutdown_connections 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f51e0078b90 0x7f51e007b0a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5200152210 0x7f52000ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52000c67a0 0x7f52001411e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 --2- 172.21.15.117:0/3077462499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52000c5490 0x7f52001f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 >> 172.21.15.117:0/3077462499 conn(0x7f520006e590 msgr2=0x7f52001e1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 shutdown_connections 2024-09-17T05:29:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:29:32.096+0000 7f5205cf2640 1 -- 172.21.15.117:0/3077462499 wait complete. 2024-09-17T05:29:32.109 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-09-17T05:29:32.109 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2024-09-17T05:29:33.056 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether build environment is sane... yes 2024-09-17T05:29:33.065 INFO:tasks.workunit.client.0.smithi117.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-17T05:29:33.066 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gawk... gawk 2024-09-17T05:29:33.084 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether make sets $(MAKE)... yes 2024-09-17T05:29:33.116 INFO:tasks.workunit.client.0.smithi117.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-17T05:29:33.117 INFO:tasks.workunit.client.0.smithi117.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-17T05:29:33.127 INFO:tasks.workunit.client.0.smithi117.stdout:checking for style of include used by make... GNU 2024-09-17T05:29:33.132 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc... gcc 2024-09-17T05:29:33.186 INFO:tasks.workunit.client.0.smithi117.stdout:checking for C compiler default output file name... a.out 2024-09-17T05:29:33.187 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether the C compiler works... yes 2024-09-17T05:29:33.190 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are cross compiling... no 2024-09-17T05:29:33.225 INFO:tasks.workunit.client.0.smithi117.stdout:checking for suffix of executables... 2024-09-17T05:29:33.258 INFO:tasks.workunit.client.0.smithi117.stdout:checking for suffix of object files... o 2024-09-17T05:29:33.288 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are using the GNU C compiler... yes 2024-09-17T05:29:33.318 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether gcc accepts -g... yes 2024-09-17T05:29:33.389 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-17T05:29:33.485 INFO:tasks.workunit.client.0.smithi117.stdout:checking dependency style of gcc... gcc3 2024-09-17T05:29:33.548 INFO:tasks.workunit.client.0.smithi117.stdout:checking how to run the C preprocessor... gcc -E 2024-09-17T05:29:33.588 INFO:tasks.workunit.client.0.smithi117.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-17T05:29:33.589 INFO:tasks.workunit.client.0.smithi117.stdout:checking for egrep... /usr/bin/grep -E 2024-09-17T05:29:33.598 INFO:tasks.workunit.client.0.smithi117.stdout:checking for AIX... no 2024-09-17T05:29:33.615 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc... (cached) gcc 2024-09-17T05:29:33.623 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-17T05:29:33.624 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-17T05:29:33.624 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-17T05:29:33.624 INFO:tasks.workunit.client.0.smithi117.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-17T05:29:33.680 INFO:tasks.workunit.client.0.smithi117.stdout:checking for main in -lm... yes 2024-09-17T05:29:33.730 INFO:tasks.workunit.client.0.smithi117.stdout:checking for main in -lpthread... yes 2024-09-17T05:29:33.929 INFO:tasks.workunit.client.0.smithi117.stdout:checking for ANSI C header files... yes 2024-09-17T05:29:33.999 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-17T05:29:34.041 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/types.h... yes 2024-09-17T05:29:34.085 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/stat.h... yes 2024-09-17T05:29:34.128 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdlib.h... yes 2024-09-17T05:29:34.172 INFO:tasks.workunit.client.0.smithi117.stdout:checking for string.h... yes 2024-09-17T05:29:34.217 INFO:tasks.workunit.client.0.smithi117.stdout:checking for memory.h... yes 2024-09-17T05:29:34.262 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strings.h... yes 2024-09-17T05:29:34.308 INFO:tasks.workunit.client.0.smithi117.stdout:checking for inttypes.h... yes 2024-09-17T05:29:34.351 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdint.h... yes 2024-09-17T05:29:34.397 INFO:tasks.workunit.client.0.smithi117.stdout:checking for unistd.h... yes 2024-09-17T05:29:34.457 INFO:tasks.workunit.client.0.smithi117.stdout:checking pthread.h usability... yes 2024-09-17T05:29:34.478 INFO:tasks.workunit.client.0.smithi117.stdout:checking pthread.h presence... yes 2024-09-17T05:29:34.478 INFO:tasks.workunit.client.0.smithi117.stdout:checking for pthread.h... yes 2024-09-17T05:29:34.531 INFO:tasks.workunit.client.0.smithi117.stdout:checking fcntl.h usability... yes 2024-09-17T05:29:34.554 INFO:tasks.workunit.client.0.smithi117.stdout:checking fcntl.h presence... yes 2024-09-17T05:29:34.554 INFO:tasks.workunit.client.0.smithi117.stdout:checking for fcntl.h... yes 2024-09-17T05:29:34.607 INFO:tasks.workunit.client.0.smithi117.stdout:checking limits.h usability... yes 2024-09-17T05:29:34.629 INFO:tasks.workunit.client.0.smithi117.stdout:checking limits.h presence... yes 2024-09-17T05:29:34.630 INFO:tasks.workunit.client.0.smithi117.stdout:checking for limits.h... yes 2024-09-17T05:29:34.643 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdint.h... (cached) yes 2024-09-17T05:29:34.690 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/time.h usability... yes 2024-09-17T05:29:34.710 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/time.h presence... yes 2024-09-17T05:29:34.710 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/time.h... yes 2024-09-17T05:29:34.723 INFO:tasks.workunit.client.0.smithi117.stdout:checking for unistd.h... (cached) yes 2024-09-17T05:29:34.767 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/vfs.h usability... yes 2024-09-17T05:29:34.786 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/vfs.h presence... yes 2024-09-17T05:29:34.787 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/vfs.h... yes 2024-09-17T05:29:34.831 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/limits.h usability... no 2024-09-17T05:29:34.851 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/limits.h presence... no 2024-09-17T05:29:34.851 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/limits.h... no 2024-09-17T05:29:34.884 INFO:tasks.workunit.client.0.smithi117.stdout:checking for an ANSI C-conforming const... yes 2024-09-17T05:29:34.916 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-17T05:29:34.917 INFO:tasks.workunit.client.0.smithi117.stdout:checking for function prototypes... yes 2024-09-17T05:29:34.953 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether setvbuf arguments are reversed... no 2024-09-17T05:29:35.024 INFO:tasks.workunit.client.0.smithi117.stdout:checking for system... yes 2024-09-17T05:29:35.084 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gettimeofday... yes 2024-09-17T05:29:35.144 INFO:tasks.workunit.client.0.smithi117.stdout:checking for mkdir... yes 2024-09-17T05:29:35.205 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strerror... yes 2024-09-17T05:29:35.265 INFO:tasks.workunit.client.0.smithi117.stdout:checking for open64... yes 2024-09-17T05:29:35.325 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stat64... yes 2024-09-17T05:29:35.384 INFO:tasks.workunit.client.0.smithi117.stdout:checking for fseeko64... yes 2024-09-17T05:29:35.444 INFO:tasks.workunit.client.0.smithi117.stdout:checking for lrand48_r... yes 2024-09-17T05:29:35.503 INFO:tasks.workunit.client.0.smithi117.stdout:checking for srand48_r... yes 2024-09-17T05:29:35.525 INFO:tasks.workunit.client.0.smithi117.stdout:configure: creating ./config.status 2024-09-17T05:29:35.621 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating Makefile 2024-09-17T05:29:35.635 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating config.h 2024-09-17T05:29:35.653 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: executing depfiles commands 2024-09-17T05:29:35.707 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: executing default commands 2024-09-17T05:29:35.752 INFO:tasks.workunit.client.0.smithi117.stderr:+ make 2024-09-17T05:29:35.763 INFO:tasks.workunit.client.0.smithi117.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-09-17T05:29:35.768 INFO:tasks.workunit.client.0.smithi117.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-17T05:29:35.769 INFO:tasks.workunit.client.0.smithi117.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-17T05:29:35.769 INFO:tasks.workunit.client.0.smithi117.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-17T05:29:35.769 INFO:tasks.workunit.client.0.smithi117.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-17T05:29:35.769 INFO:tasks.workunit.client.0.smithi117.stderr: any GNU archive site. 2024-09-17T05:29:35.771 INFO:tasks.workunit.client.0.smithi117.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-17T05:29:35.772 INFO:tasks.workunit.client.0.smithi117.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-17T05:29:35.774 INFO:tasks.workunit.client.0.smithi117.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-17T05:29:35.774 INFO:tasks.workunit.client.0.smithi117.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-17T05:29:35.774 INFO:tasks.workunit.client.0.smithi117.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-17T05:29:35.774 INFO:tasks.workunit.client.0.smithi117.stderr: Grab them from any GNU archive site. 2024-09-17T05:29:35.827 INFO:tasks.workunit.client.0.smithi117.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-09-17T05:29:36.158 INFO:tasks.workunit.client.0.smithi117.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-17T05:29:36.158 INFO:tasks.workunit.client.0.smithi117.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-17T05:29:36.158 INFO:tasks.workunit.client.0.smithi117.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-17T05:29:36.158 INFO:tasks.workunit.client.0.smithi117.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-17T05:29:36.373 INFO:tasks.workunit.client.0.smithi117.stdout:/bin/sh ./config.status --recheck 2024-09-17T05:29:36.433 INFO:tasks.workunit.client.0.smithi117.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-09-17T05:29:36.530 INFO:tasks.workunit.client.0.smithi117.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-17T05:29:37.537 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether build environment is sane... yes 2024-09-17T05:29:37.541 INFO:tasks.workunit.client.0.smithi117.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-17T05:29:37.541 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gawk... gawk 2024-09-17T05:29:37.564 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether make sets $(MAKE)... yes 2024-09-17T05:29:37.593 INFO:tasks.workunit.client.0.smithi117.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-17T05:29:37.593 INFO:tasks.workunit.client.0.smithi117.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-17T05:29:37.604 INFO:tasks.workunit.client.0.smithi117.stdout:checking for style of include used by make... GNU 2024-09-17T05:29:37.608 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc... gcc 2024-09-17T05:29:37.710 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether the C compiler works... yes 2024-09-17T05:29:37.710 INFO:tasks.workunit.client.0.smithi117.stdout:checking for C compiler default output file name... a.out 2024-09-17T05:29:37.746 INFO:tasks.workunit.client.0.smithi117.stdout:checking for suffix of executables... 2024-09-17T05:29:37.782 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are cross compiling... no 2024-09-17T05:29:37.817 INFO:tasks.workunit.client.0.smithi117.stdout:checking for suffix of object files... o 2024-09-17T05:29:37.840 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are using the GNU C compiler... yes 2024-09-17T05:29:37.863 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether gcc accepts -g... yes 2024-09-17T05:29:37.902 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-17T05:29:37.994 INFO:tasks.workunit.client.0.smithi117.stdout:checking dependency style of gcc... gcc3 2024-09-17T05:29:38.042 INFO:tasks.workunit.client.0.smithi117.stdout:checking how to run the C preprocessor... gcc -E 2024-09-17T05:29:38.092 INFO:tasks.workunit.client.0.smithi117.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-17T05:29:38.093 INFO:tasks.workunit.client.0.smithi117.stdout:checking for egrep... /usr/bin/grep -E 2024-09-17T05:29:38.213 INFO:tasks.workunit.client.0.smithi117.stdout:checking for ANSI C header files... yes 2024-09-17T05:29:38.247 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/types.h... yes 2024-09-17T05:29:38.286 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/stat.h... yes 2024-09-17T05:29:38.326 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdlib.h... yes 2024-09-17T05:29:38.363 INFO:tasks.workunit.client.0.smithi117.stdout:checking for string.h... yes 2024-09-17T05:29:38.401 INFO:tasks.workunit.client.0.smithi117.stdout:checking for memory.h... yes 2024-09-17T05:29:38.438 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strings.h... yes 2024-09-17T05:29:38.476 INFO:tasks.workunit.client.0.smithi117.stdout:checking for inttypes.h... yes 2024-09-17T05:29:38.517 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdint.h... yes 2024-09-17T05:29:38.557 INFO:tasks.workunit.client.0.smithi117.stdout:checking for unistd.h... yes 2024-09-17T05:29:38.603 INFO:tasks.workunit.client.0.smithi117.stdout:checking minix/config.h usability... no 2024-09-17T05:29:38.630 INFO:tasks.workunit.client.0.smithi117.stdout:checking minix/config.h presence... no 2024-09-17T05:29:38.630 INFO:tasks.workunit.client.0.smithi117.stdout:checking for minix/config.h... no 2024-09-17T05:29:38.664 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-17T05:29:38.664 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc... (cached) gcc 2024-09-17T05:29:38.700 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-17T05:29:38.701 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-17T05:29:38.701 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-17T05:29:38.701 INFO:tasks.workunit.client.0.smithi117.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-17T05:29:38.744 INFO:tasks.workunit.client.0.smithi117.stdout:checking for main in -lm... yes 2024-09-17T05:29:38.786 INFO:tasks.workunit.client.0.smithi117.stdout:checking for main in -lpthread... yes 2024-09-17T05:29:38.787 INFO:tasks.workunit.client.0.smithi117.stdout:checking for ANSI C header files... (cached) yes 2024-09-17T05:29:38.845 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-17T05:29:38.901 INFO:tasks.workunit.client.0.smithi117.stdout:checking pthread.h usability... yes 2024-09-17T05:29:38.919 INFO:tasks.workunit.client.0.smithi117.stdout:checking pthread.h presence... yes 2024-09-17T05:29:38.919 INFO:tasks.workunit.client.0.smithi117.stdout:checking for pthread.h... yes 2024-09-17T05:29:38.986 INFO:tasks.workunit.client.0.smithi117.stdout:checking fcntl.h usability... yes 2024-09-17T05:29:39.003 INFO:tasks.workunit.client.0.smithi117.stdout:checking fcntl.h presence... yes 2024-09-17T05:29:39.003 INFO:tasks.workunit.client.0.smithi117.stdout:checking for fcntl.h... yes 2024-09-17T05:29:39.091 INFO:tasks.workunit.client.0.smithi117.stdout:checking limits.h usability... yes 2024-09-17T05:29:39.107 INFO:tasks.workunit.client.0.smithi117.stdout:checking limits.h presence... yes 2024-09-17T05:29:39.108 INFO:tasks.workunit.client.0.smithi117.stdout:checking for limits.h... yes 2024-09-17T05:29:39.120 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdint.h... (cached) yes 2024-09-17T05:29:39.225 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/time.h usability... yes 2024-09-17T05:29:39.242 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/time.h presence... yes 2024-09-17T05:29:39.242 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/time.h... yes 2024-09-17T05:29:39.254 INFO:tasks.workunit.client.0.smithi117.stdout:checking for unistd.h... (cached) yes 2024-09-17T05:29:39.302 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/vfs.h usability... yes 2024-09-17T05:29:39.318 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/vfs.h presence... yes 2024-09-17T05:29:39.319 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/vfs.h... yes 2024-09-17T05:29:39.418 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/limits.h usability... no 2024-09-17T05:29:39.449 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/limits.h presence... no 2024-09-17T05:29:39.449 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/limits.h... no 2024-09-17T05:29:39.498 INFO:tasks.workunit.client.0.smithi117.stdout:checking for an ANSI C-conforming const... yes 2024-09-17T05:29:39.549 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-17T05:29:39.675 INFO:tasks.workunit.client.0.smithi117.stdout:checking for system... yes 2024-09-17T05:29:39.768 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gettimeofday... yes 2024-09-17T05:29:39.835 INFO:tasks.workunit.client.0.smithi117.stdout:checking for mkdir... yes 2024-09-17T05:29:39.918 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strerror... yes 2024-09-17T05:29:40.043 INFO:tasks.workunit.client.0.smithi117.stdout:checking for open64... yes 2024-09-17T05:29:40.100 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stat64... yes 2024-09-17T05:29:40.156 INFO:tasks.workunit.client.0.smithi117.stdout:checking for fseeko64... yes 2024-09-17T05:29:40.212 INFO:tasks.workunit.client.0.smithi117.stdout:checking for lrand48_r... yes 2024-09-17T05:29:40.268 INFO:tasks.workunit.client.0.smithi117.stdout:checking for srand48_r... yes 2024-09-17T05:29:40.289 INFO:tasks.workunit.client.0.smithi117.stdout:configure: creating ./config.status 2024-09-17T05:29:40.349 INFO:tasks.workunit.client.0.smithi117.stdout: /bin/sh ./config.status 2024-09-17T05:29:40.402 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating Makefile 2024-09-17T05:29:40.415 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating config.h 2024-09-17T05:29:40.437 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: executing depfiles commands 2024-09-17T05:29:40.447 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: executing default commands 2024-09-17T05:29:40.472 INFO:tasks.workunit.client.0.smithi117.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-09-17T05:29:40.626 INFO:tasks.workunit.client.0.smithi117.stdout:rm -f stamp-h1 2024-09-17T05:29:40.629 INFO:tasks.workunit.client.0.smithi117.stdout:touch config.h.in 2024-09-17T05:29:40.630 INFO:tasks.workunit.client.0.smithi117.stdout:cd . && /bin/sh ./config.status config.h 2024-09-17T05:29:40.675 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating config.h 2024-09-17T05:29:40.703 INFO:tasks.workunit.client.0.smithi117.stdout:make all-am 2024-09-17T05:29:40.709 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-17T05:29:40.709 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-17T05:29:40.889 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-17T05:29:40.893 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-17T05:29:40.988 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-17T05:29:40.992 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-17T05:29:41.083 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-17T05:29:41.087 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-17T05:29:41.173 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-17T05:29:41.176 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-17T05:29:41.304 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-17T05:29:41.307 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-17T05:29:41.353 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-17T05:29:41.357 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-17T05:29:41.398 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-17T05:29:41.400 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-17T05:29:41.445 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-17T05:29:41.447 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-17T05:29:41.642 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-17T05:29:41.645 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-17T05:29:41.777 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-17T05:29:41.782 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-17T05:29:41.940 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-17T05:29:41.943 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-17T05:29:42.009 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-17T05:29:42.013 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-17T05:29:42.092 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-17T05:29:42.095 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-17T05:29:42.197 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-17T05:29:42.201 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-17T05:29:42.229 INFO:tasks.workunit.client.0.smithi117.stderr:In file included from parser.h:22, 2024-09-17T05:29:42.229 INFO:tasks.workunit.client.0.smithi117.stderr: from parser.c:25: 2024-09-17T05:29:42.229 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c: In function ‘init_filesys’: 2024-09-17T05:29:42.229 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.229 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.229 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.230 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.231 INFO:tasks.workunit.client.0.smithi117.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T05:29:42.232 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.233 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c: In function ‘init_tg_stats’: 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.234 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.235 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.236 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T05:29:42.237 INFO:tasks.workunit.client.0.smithi117.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr: | ^ 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~ 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr: 36 | container_of(ptr, type, member) 2024-09-17T05:29:42.238 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~ 2024-09-17T05:29:42.239 INFO:tasks.workunit.client.0.smithi117.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-17T05:29:42.239 INFO:tasks.workunit.client.0.smithi117.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T05:29:42.239 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~ 2024-09-17T05:29:42.239 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-17T05:29:42.239 INFO:tasks.workunit.client.0.smithi117.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-17T05:29:42.239 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c: In function ‘get_optstr’: 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:200:14: note: source object declared here 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr: 200 | char temp[BUFSIZE]; 2024-09-17T05:29:42.278 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~ 2024-09-17T05:29:42.284 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c: In function ‘size64_convert’: 2024-09-17T05:29:42.285 INFO:tasks.workunit.client.0.smithi117.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-17T05:29:42.285 INFO:tasks.workunit.client.0.smithi117.stderr: 119 | memcpy(unit, "\0", 3); 2024-09-17T05:29:42.285 INFO:tasks.workunit.client.0.smithi117.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-09-17T05:29:42.566 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-17T05:29:42.570 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-17T05:29:42.644 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-17T05:29:42.647 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-17T05:29:42.737 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-17T05:29:42.741 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-17T05:29:42.771 INFO:tasks.workunit.client.0.smithi117.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-17T05:29:42.775 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-17T05:29:42.807 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-17T05:29:42.808 INFO:tasks.workunit.client.0.smithi117.stderr:+ cd .. 2024-09-17T05:29:42.808 INFO:tasks.workunit.client.0.smithi117.stderr:+ mkdir tmp 2024-09-17T05:29:42.810 INFO:tasks.workunit.client.0.smithi117.stderr:+ cd tmp 2024-09-17T05:29:42.810 INFO:tasks.workunit.client.0.smithi117.stderr:+ for f in $mydir/*.ffsb 2024-09-17T05:29:42.810 INFO:tasks.workunit.client.0.smithi117.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-09-17T05:29:42.810 INFO:tasks.workunit.client.0.smithi117.stdout:FFSB version 6.0-RC2 started 2024-09-17T05:29:42.811 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.813 INFO:tasks.workunit.client.0.smithi117.stdout:benchmark time = 300 2024-09-17T05:29:42.813 INFO:tasks.workunit.client.0.smithi117.stdout:ThreadGroup 0 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout:================ 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: num_threads = 32 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: read_random = off 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: read_size = 0 (0B) 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: read_blocksize = 0 (0B) 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: read_skip = off 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: read_skipsize = 0 (0B) 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: write_random = on 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: write_size = 5242880 (5MB) 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: fsync_file = 0 2024-09-17T05:29:42.814 INFO:tasks.workunit.client.0.smithi117.stdout: write_blocksize = 4096 (4KB) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: wait time = 0 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: op weights 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: read = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: readall = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: write = 1 (100.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: create = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: append = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: delete = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: metaop = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: createdir = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: stat = 0 (0.00%) 2024-09-17T05:29:42.815 INFO:tasks.workunit.client.0.smithi117.stdout: writeall = 0 (0.00%) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: writeall_fsync = 0 (0.00%) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: open_close = 0 (0.00%) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: write_fsync = 0 (0.00%) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: create_fsync = 0 (0.00%) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: append_fsync = 0 (0.00%) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout:FileSystem . 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout:========== 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: num_dirs = 0 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: starting files = 128 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: min file size = 104857600 (100MB) 2024-09-17T05:29:42.816 INFO:tasks.workunit.client.0.smithi117.stdout: max file size = 104857600 (100MB) 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: directio = off 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: alignedio = on 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: bufferedio = off 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: aging is off 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: current utilization = 0.03% 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:29:42.817 INFO:tasks.workunit.client.0.smithi117.stdout:checking existing fs: . 2024-09-17T05:29:42.818 INFO:tasks.workunit.client.0.smithi117.stdout:recreating new fileset 2024-09-17T05:29:42.818 INFO:tasks.workunit.client.0.smithi117.stderr:opendir: No such file or directory 2024-09-17T05:29:52.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:29:52.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:29:55.442 INFO:tasks.workunit.client.0.smithi117.stdout:fs setup took 12 secs 2024-09-17T05:30:01.704 INFO:tasks.workunit.client.0.smithi117.stdout:Syncing()...6 sec 2024-09-17T05:30:01.705 INFO:tasks.workunit.client.0.smithi117.stdout:Starting Actual Benchmark At: Tue Sep 17 05:30:01 2024 2024-09-17T05:30:01.705 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:30:22.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:30:22.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:30:53.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:30:53.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:31:21.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:23.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:31:23.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:31:27.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.837+0000 7fae0090f640 1 -- 172.21.15.117:0/560528919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0c7fa0 msgr2=0x7fadfc161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:27.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.837+0000 7fae0090f640 1 --2- 172.21.15.117:0/560528919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0c7fa0 0x7fadfc161dd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fadfc073300 tx=0x7fadf402ad70 comp rx=0 tx=0).stop 2024-09-17T05:31:27.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.838+0000 7fae0090f640 1 -- 172.21.15.117:0/560528919 shutdown_connections 2024-09-17T05:31:27.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.838+0000 7fae0090f640 1 --2- 172.21.15.117:0/560528919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0c7fa0 0x7fadfc161dd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:27.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.838+0000 7fae0090f640 1 --2- 172.21.15.117:0/560528919 >> v2:172.21.15.119:3300/0 conn(0x7fadfc1516d0 0x7fadfc157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:27.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.838+0000 7fae0090f640 1 -- 172.21.15.117:0/560528919 >> 172.21.15.117:0/560528919 conn(0x7fadfc06cda0 msgr2=0x7fadfc06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:27.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.840+0000 7fae0090f640 1 -- 172.21.15.117:0/560528919 shutdown_connections 2024-09-17T05:31:27.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.840+0000 7fae0090f640 1 -- 172.21.15.117:0/560528919 wait complete. 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.843+0000 7fae0090f640 1 Processor -- start 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.844+0000 7fae0090f640 1 -- start start 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fae0090f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fadfc0c7fa0 0x7fadfc13f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fae0090f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 0x7fadfc074a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fae0090f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadfc1516d0 0x7fadfc1451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fae0090f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fadfc158330 con 0x7fadfc0cc910 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fae0090f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fadfc1575f0 con 0x7fadfc1516d0 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fae0090f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fadfc0c83b0 con 0x7fadfc0c7fa0 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fadfaffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 0x7fadfc074a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:27.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fadfaffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 0x7fadfc074a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56374/0 (socket says 172.21.15.117:56374) 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.846+0000 7fadfaffd640 1 -- 172.21.15.117:0/4095244340 learned_addr learned my addr 172.21.15.117:0/4095244340 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfbfff640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadfc1516d0 0x7fadfc1451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfaffd640 1 -- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fadfc0c7fa0 msgr2=0x7fadfc13f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfaffd640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fadfc0c7fa0 0x7fadfc13f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfaffd640 1 -- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadfc1516d0 msgr2=0x7fadfc1451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfaffd640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadfc1516d0 0x7fadfc1451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:27.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfaffd640 1 -- 172.21.15.117:0/4095244340 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc13f860 con 0x7fadfc0cc910 2024-09-17T05:31:27.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.847+0000 7fadfbfff640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadfc1516d0 0x7fadfc1451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:31:27.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.848+0000 7fadfaffd640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 0x7fadfc074a90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fadec00b3f0 tx=0x7fadec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:27.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.848+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec015020 con 0x7fadfc0cc910 2024-09-17T05:31:27.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.848+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadec011920 con 0x7fadfc0cc910 2024-09-17T05:31:27.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.848+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadec011d50 con 0x7fadfc0cc910 2024-09-17T05:31:27.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.850+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadfc142130 con 0x7fadfc0cc910 2024-09-17T05:31:27.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.850+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadfc1423f0 con 0x7fadfc0cc910 2024-09-17T05:31:27.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.855+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadfc0b29e0 con 0x7fadfc0cc910 2024-09-17T05:31:27.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.856+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fadec018020 con 0x7fadfc0cc910 2024-09-17T05:31:27.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.857+0000 7fadebfff640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fade0078c90 0x7fade007b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:27.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.857+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fadec09a6e0 con 0x7fadfc0cc910 2024-09-17T05:31:27.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.863+0000 7fadfb7fe640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fade0078c90 0x7fade007b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:27.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.865+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fadec01f020 con 0x7fadfc0cc910 2024-09-17T05:31:27.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:27.865+0000 7fadfb7fe640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fade0078c90 0x7fade007b1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009650 tx=0x7fadf0006ba0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:28.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.327+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadfc0cc000 con 0x7fadfc0cc910 2024-09-17T05:31:28.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.328+0000 7fadebfff640 1 -- 172.21.15.117:0/4095244340 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7fadec062840 con 0x7fadfc0cc910 2024-09-17T05:31:28.329 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:28.329 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:28.329 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.330+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fade0078c90 msgr2=0x7fade007b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.330+0000 7fae0090f640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fade0078c90 0x7fade007b1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009650 tx=0x7fadf0006ba0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 msgr2=0x7fadfc074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 0x7fadfc074a90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fadec00b3f0 tx=0x7fadec00b8b0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 shutdown_connections 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fade0078c90 0x7fade007b1a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadfc1516d0 0x7fadfc1451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadfc0cc910 0x7fadfc074a90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 --2- 172.21.15.117:0/4095244340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fadfc0c7fa0 0x7fadfc13f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 >> 172.21.15.117:0/4095244340 conn(0x7fadfc06cda0 msgr2=0x7fadfc1552c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 shutdown_connections 2024-09-17T05:31:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.331+0000 7fae0090f640 1 -- 172.21.15.117:0/4095244340 wait complete. 2024-09-17T05:31:28.354 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2024-09-17T05:31:28.354 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:31:28.354 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:31:28.368 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:31:28.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:28.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.618+0000 7fa980849640 1 -- 172.21.15.117:0/3691062621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740e39d0 msgr2=0x7fa9740e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:28.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.618+0000 7fa980849640 1 --2- 172.21.15.117:0/3691062621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740e39d0 0x7fa9740e5df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009820 tx=0x7fa96c02a4f0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.618+0000 7fa980849640 1 -- 172.21.15.117:0/3691062621 shutdown_connections 2024-09-17T05:31:28.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.618+0000 7fa980849640 1 --2- 172.21.15.117:0/3691062621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740e39d0 0x7fa9740e5df0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.618+0000 7fa980849640 1 --2- 172.21.15.117:0/3691062621 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa9740db160 0x7fa9740db570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.618+0000 7fa980849640 1 -- 172.21.15.117:0/3691062621 >> 172.21.15.117:0/3691062621 conn(0x7fa97404fe30 msgr2=0x7fa974050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:28.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.620+0000 7fa980849640 1 -- 172.21.15.117:0/3691062621 shutdown_connections 2024-09-17T05:31:28.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.621+0000 7fa980849640 1 -- 172.21.15.117:0/3691062621 wait complete. 2024-09-17T05:31:28.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.624+0000 7fa980849640 1 Processor -- start 2024-09-17T05:31:28.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.624+0000 7fa980849640 1 -- start start 2024-09-17T05:31:28.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.627+0000 7fa980849640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740d5400 0x7fa9740d5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:28.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.627+0000 7fa980849640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa9740db160 0x7fa974048ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:28.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.627+0000 7fa980849640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 0x7fa97404f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:28.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.627+0000 7fa980849640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa9740354c0 con 0x7fa9740e39d0 2024-09-17T05:31:28.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.627+0000 7fa980849640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa9740351e0 con 0x7fa9740d5400 2024-09-17T05:31:28.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.627+0000 7fa980849640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa974033540 con 0x7fa9740db160 2024-09-17T05:31:28.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.630+0000 7fa97b59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 0x7fa97404f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:28.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.630+0000 7fa97b59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 0x7fa97404f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56386/0 (socket says 172.21.15.117:56386) 2024-09-17T05:31:28.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.630+0000 7fa97b59e640 1 -- 172.21.15.117:0/916321823 learned_addr learned my addr 172.21.15.117:0/916321823 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.630+0000 7fa97ad9d640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740d5400 0x7fa9740d5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.630+0000 7fa97a59c640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa9740db160 0x7fa974048ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.630+0000 7fa97b59e640 1 -- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa9740db160 msgr2=0x7fa974048ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.631+0000 7fa97b59e640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa9740db160 0x7fa974048ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.631+0000 7fa97b59e640 1 -- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740d5400 msgr2=0x7fa9740d5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.631+0000 7fa97b59e640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740d5400 0x7fa9740d5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.631+0000 7fa97b59e640 1 -- 172.21.15.117:0/916321823 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa974049430 con 0x7fa9740e39d0 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.631+0000 7fa97ad9d640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740d5400 0x7fa9740d5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.631+0000 7fa97b59e640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 0x7fa97404f390 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa96c02aae0 tx=0x7fa96c004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.632+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c031950 con 0x7fa9740e39d0 2024-09-17T05:31:28.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.632+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa96c046030 con 0x7fa9740e39d0 2024-09-17T05:31:28.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.633+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa974047440 con 0x7fa9740e39d0 2024-09-17T05:31:28.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.634+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9740d0cb0 con 0x7fa9740e39d0 2024-09-17T05:31:28.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.636+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c031e60 con 0x7fa9740e39d0 2024-09-17T05:31:28.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.636+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa96c02ad10 con 0x7fa9740e39d0 2024-09-17T05:31:28.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.637+0000 7fa9637fe640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa968078c40 0x7fa96807b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:28.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.637+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fa96c031750 con 0x7fa9740e39d0 2024-09-17T05:31:28.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.638+0000 7fa97ad9d640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa968078c40 0x7fa96807b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:28.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.638+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa95c005290 con 0x7fa9740e39d0 2024-09-17T05:31:28.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.644+0000 7fa97ad9d640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa968078c40 0x7fa96807b150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa97c066830 tx=0x7fa97c070040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:28.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:28.648+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa96c08b7b0 con 0x7fa9740e39d0 2024-09-17T05:31:29.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.101+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa95c005850 con 0x7fa9740e39d0 2024-09-17T05:31:29.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.101+0000 7fa9637fe640 1 -- 172.21.15.117:0/916321823 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7fa96c08b0b0 con 0x7fa9740e39d0 2024-09-17T05:31:29.104 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:29.104 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:29.104 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.105+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa968078c40 msgr2=0x7fa96807b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.105+0000 7fa980849640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa968078c40 0x7fa96807b150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa97c066830 tx=0x7fa97c070040 comp rx=0 tx=0).stop 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.105+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 msgr2=0x7fa97404f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.105+0000 7fa980849640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 0x7fa97404f390 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa96c02aae0 tx=0x7fa96c004a90 comp rx=0 tx=0).stop 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.106+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 shutdown_connections 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.106+0000 7fa980849640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa968078c40 0x7fa96807b150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.106+0000 7fa980849640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740e39d0 0x7fa97404f390 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:29.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.106+0000 7fa980849640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa9740db160 0x7fa974048ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:29.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.106+0000 7fa980849640 1 --2- 172.21.15.117:0/916321823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9740d5400 0x7fa9740d5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:29.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.106+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 >> 172.21.15.117:0/916321823 conn(0x7fa97404fe30 msgr2=0x7fa9740d9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.110+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 shutdown_connections 2024-09-17T05:31:29.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:29.110+0000 7fa980849640 1 -- 172.21.15.117:0/916321823 wait complete. 2024-09-17T05:31:29.131 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:31.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:31.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.345+0000 7fe16b89b640 1 -- 172.21.15.117:0/2782487359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c5ad0 msgr2=0x7fe1640c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:31.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.345+0000 7fe16b89b640 1 --2- 172.21.15.117:0/2782487359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c5ad0 0x7fe1640c5ee0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe16415b560 tx=0x7fe15802a700 comp rx=0 tx=0).stop 2024-09-17T05:31:31.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.348+0000 7fe16b89b640 1 -- 172.21.15.117:0/2782487359 shutdown_connections 2024-09-17T05:31:31.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.348+0000 7fe16b89b640 1 --2- 172.21.15.117:0/2782487359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c5ad0 0x7fe1640c5ee0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.348+0000 7fe16b89b640 1 -- 172.21.15.117:0/2782487359 >> 172.21.15.117:0/2782487359 conn(0x7fe16406e020 msgr2=0x7fe16406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:31.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.350+0000 7fe16b89b640 1 -- 172.21.15.117:0/2782487359 shutdown_connections 2024-09-17T05:31:31.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.352+0000 7fe16b89b640 1 -- 172.21.15.117:0/2782487359 wait complete. 2024-09-17T05:31:31.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.355+0000 7fe16b89b640 1 Processor -- start 2024-09-17T05:31:31.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.356+0000 7fe16b89b640 1 -- start start 2024-09-17T05:31:31.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.358+0000 7fe16b89b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1640c5ad0 0x7fe1640d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:31.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.358+0000 7fe16b89b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c7120 0x7fe16414fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:31.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.358+0000 7fe16b89b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe16415e870 0x7fe1640d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:31.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe16b89b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe16415bf60 con 0x7fe1640c7120 2024-09-17T05:31:31.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe168e36640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c7120 0x7fe16414fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:31.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe169e38640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe16415e870 0x7fe1640d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:31.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe169e38640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe16415e870 0x7fe1640d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:50548/0 (socket says 172.21.15.117:50548) 2024-09-17T05:31:31.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe169e38640 1 -- 172.21.15.117:0/3563409957 learned_addr learned my addr 172.21.15.117:0/3563409957 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe16b89b640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe1640cc210 con 0x7fe1640c5ad0 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe16b89b640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe16415bbf0 con 0x7fe16415e870 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.359+0000 7fe169637640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1640c5ad0 0x7fe1640d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.360+0000 7fe168e36640 1 -- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe16415e870 msgr2=0x7fe1640d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.360+0000 7fe168e36640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe16415e870 0x7fe1640d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.360+0000 7fe168e36640 1 -- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1640c5ad0 msgr2=0x7fe1640d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.360+0000 7fe168e36640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1640c5ad0 0x7fe1640d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.360+0000 7fe168e36640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16414a7f0 con 0x7fe1640c7120 2024-09-17T05:31:31.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.360+0000 7fe168e36640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c7120 0x7fe16414fa60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe158031000 tx=0x7fe158003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:31.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.361+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15803e030 con 0x7fe1640c7120 2024-09-17T05:31:31.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.361+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1580047a0 con 0x7fe1640c7120 2024-09-17T05:31:31.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.364+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158035040 con 0x7fe1640c7120 2024-09-17T05:31:31.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.364+0000 7fe16b89b640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1640d0920 con 0x7fe1640c7120 2024-09-17T05:31:31.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.364+0000 7fe16b89b640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe16415da90 con 0x7fe1640c7120 2024-09-17T05:31:31.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.366+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe16406af90 con 0x7fe1640c7120 2024-09-17T05:31:31.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.369+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe158041030 con 0x7fe1640c7120 2024-09-17T05:31:31.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.370+0000 7fe161ffb640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe150078c60 0x7fe15007b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:31.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.370+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fe1580bbb30 con 0x7fe1640c7120 2024-09-17T05:31:31.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.382+0000 7fe169637640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe150078c60 0x7fe15007b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:31.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.386+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe16406af90 con 0x7fe1640c7120 2024-09-17T05:31:31.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.396+0000 7fe169637640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe150078c60 0x7fe15007b170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe154000f60 tx=0x7fe154009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:31.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.819+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1640c5ee0 con 0x7fe1640c7120 2024-09-17T05:31:31.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.821+0000 7fe161ffb640 1 -- 172.21.15.117:0/3563409957 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7fe158083650 con 0x7fe1640c7120 2024-09-17T05:31:31.825 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:31.825 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:31.825 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:31.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.826+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe150078c60 msgr2=0x7fe15007b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.826+0000 7fe14b7fe640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe150078c60 0x7fe15007b170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe154000f60 tx=0x7fe154009040 comp rx=0 tx=0).stop 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.826+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c7120 msgr2=0x7fe16414fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.826+0000 7fe14b7fe640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c7120 0x7fe16414fa60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe158031000 tx=0x7fe158003960 comp rx=0 tx=0).stop 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 shutdown_connections 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe150078c60 0x7fe15007b170 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe16415e870 0x7fe1640d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1640c7120 0x7fe16414fa60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 --2- 172.21.15.117:0/3563409957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1640c5ad0 0x7fe1640d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 >> 172.21.15.117:0/3563409957 conn(0x7fe16406e020 msgr2=0x7fe1641545f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 shutdown_connections 2024-09-17T05:31:31.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:31.827+0000 7fe14b7fe640 1 -- 172.21.15.117:0/3563409957 wait complete. 2024-09-17T05:31:31.854 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:33.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 -- 172.21.15.117:0/3219005793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8558074750 msgr2=0x7f8558074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 --2- 172.21.15.117:0/3219005793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8558074750 0x7f8558074b60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f85580719c0 tx=0x7f855002a7e0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 -- 172.21.15.117:0/3219005793 shutdown_connections 2024-09-17T05:31:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 --2- 172.21.15.117:0/3219005793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8558074750 0x7f8558074b60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 --2- 172.21.15.117:0/3219005793 >> v2:172.21.15.119:3300/0 conn(0x7f85580c92c0 0x7f85580c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 --2- 172.21.15.117:0/3219005793 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f855808c4c0 0x7f85580c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.301+0000 7f855f070640 1 -- 172.21.15.117:0/3219005793 >> 172.21.15.117:0/3219005793 conn(0x7f855806d190 msgr2=0x7f855806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.303+0000 7f855f070640 1 -- 172.21.15.117:0/3219005793 shutdown_connections 2024-09-17T05:31:34.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.303+0000 7f855f070640 1 -- 172.21.15.117:0/3219005793 wait complete. 2024-09-17T05:31:34.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.305+0000 7f855f070640 1 Processor -- start 2024-09-17T05:31:34.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.306+0000 7f855f070640 1 -- start start 2024-09-17T05:31:34.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855f070640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 0x7f85581f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:34.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855f070640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f855808c4c0 0x7f855816a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:34.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855f070640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85580c92c0 0x7f855815ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:34.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855f070640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f85580c8b80 con 0x7f855808c4c0 2024-09-17T05:31:34.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855f070640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f855816b5d0 con 0x7f8558074750 2024-09-17T05:31:34.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855f070640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f85580c8d00 con 0x7f85580c92c0 2024-09-17T05:31:34.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855ce0c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 0x7f85581f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855ce0c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 0x7f85581f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:42756/0 (socket says 172.21.15.117:42756) 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.308+0000 7f855ce0c640 1 -- 172.21.15.117:0/1742447215 learned_addr learned my addr 172.21.15.117:0/1742447215 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855d60d640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85580c92c0 0x7f855815ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855ce0c640 1 -- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85580c92c0 msgr2=0x7f855815ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855ce0c640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85580c92c0 0x7f855815ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855ce0c640 1 -- 172.21.15.117:0/1742447215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f855808c4c0 msgr2=0x7f855816a570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855ce0c640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f855808c4c0 0x7f855816a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855ce0c640 1 -- 172.21.15.117:0/1742447215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85581f3d00 con 0x7f8558074750 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.309+0000 7f855ce0c640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 0x7f85581f37c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f854800cce0 tx=0x7f854800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.310+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854800f450 con 0x7f8558074750 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.310+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f854800f6e0 con 0x7f8558074750 2024-09-17T05:31:34.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.310+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854800faf0 con 0x7f8558074750 2024-09-17T05:31:34.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.310+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8558165470 con 0x7f8558074750 2024-09-17T05:31:34.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.310+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85581663d0 con 0x7f8558074750 2024-09-17T05:31:34.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.312+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853c005290 con 0x7f8558074750 2024-09-17T05:31:34.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.313+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f854800fc90 con 0x7f8558074750 2024-09-17T05:31:34.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.314+0000 7f85557fa640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8538078b90 0x7f853807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:34.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.314+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f854809bbc0 con 0x7f8558074750 2024-09-17T05:31:34.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.317+0000 7f8557fff640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8538078b90 0x7f853807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:34.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.323+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8548014030 con 0x7f8558074750 2024-09-17T05:31:34.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.324+0000 7f8557fff640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8538078b90 0x7f853807b0a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8540005dc0 tx=0x7f8540005d30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:34.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.792+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f853c005580 con 0x7f8558074750 2024-09-17T05:31:34.796 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:34.797 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:34.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.795+0000 7f85557fa640 1 -- 172.21.15.117:0/1742447215 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7f8548063ba0 con 0x7f8558074750 2024-09-17T05:31:34.797 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:34.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.799+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8538078b90 msgr2=0x7f853807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:34.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.799+0000 7f855f070640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8538078b90 0x7f853807b0a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8540005dc0 tx=0x7f8540005d30 comp rx=0 tx=0).stop 2024-09-17T05:31:34.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.799+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 msgr2=0x7f85581f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:34.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.802+0000 7f855f070640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 0x7f85581f37c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f854800cce0 tx=0x7f854800a620 comp rx=0 tx=0).stop 2024-09-17T05:31:34.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.803+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 shutdown_connections 2024-09-17T05:31:34.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.803+0000 7f855f070640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8538078b90 0x7f853807b0a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.803+0000 7f855f070640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85580c92c0 0x7f855815ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.803+0000 7f855f070640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f855808c4c0 0x7f855816a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.803+0000 7f855f070640 1 --2- 172.21.15.117:0/1742447215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8558074750 0x7f85581f37c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:34.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.803+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 >> 172.21.15.117:0/1742447215 conn(0x7f855806d190 msgr2=0x7f85581680b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:34.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.805+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 shutdown_connections 2024-09-17T05:31:34.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:34.805+0000 7f855f070640 1 -- 172.21.15.117:0/1742447215 wait complete. 2024-09-17T05:31:34.826 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:36.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:37.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.043+0000 7f36f359e640 1 -- 172.21.15.117:0/1856651130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36dc0053f0 msgr2=0x7f36dc00ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:37.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.043+0000 7f36f359e640 1 --2- 172.21.15.117:0/1856651130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36dc0053f0 0x7f36dc00ba80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0043b0 tx=0x7f36dc005830 comp rx=0 tx=0).stop 2024-09-17T05:31:37.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.044+0000 7f36f359e640 1 -- 172.21.15.117:0/1856651130 shutdown_connections 2024-09-17T05:31:37.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.044+0000 7f36f359e640 1 --2- 172.21.15.117:0/1856651130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36dc0053f0 0x7f36dc00ba80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.044+0000 7f36f359e640 1 -- 172.21.15.117:0/1856651130 >> 172.21.15.117:0/1856651130 conn(0x7f36f40c8d10 msgr2=0x7f36f40c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:37.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.046+0000 7f36f359e640 1 -- 172.21.15.117:0/1856651130 shutdown_connections 2024-09-17T05:31:37.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.051+0000 7f36f359e640 1 -- 172.21.15.117:0/1856651130 wait complete. 2024-09-17T05:31:37.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.054+0000 7f36f359e640 1 Processor -- start 2024-09-17T05:31:37.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.057+0000 7f36f359e640 1 -- start start 2024-09-17T05:31:37.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.060+0000 7f36f359e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 0x7f36f415c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:37.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.060+0000 7f36f359e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f40c2cf0 0x7f36f40c0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 0x7f36f415c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 0x7f36f415c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:57596/0 (socket says 172.21.15.117:57596) 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f259c640 1 -- 172.21.15.117:0/1522454267 learned_addr learned my addr 172.21.15.117:0/1522454267 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36f40c5c90 0x7f36f414ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36f4073aa0 con 0x7f36f40c2cf0 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f2d9d640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36f40c5c90 0x7f36f414ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f36f40737c0 con 0x7f36f40c5c90 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f36f4073d80 con 0x7f36f415f2b0 2024-09-17T05:31:37.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.061+0000 7f36f1d9b640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f40c2cf0 0x7f36f40c0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:37.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f1d9b640 1 -- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 msgr2=0x7f36f415c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:37.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f1d9b640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 0x7f36f415c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f1d9b640 1 -- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36f40c5c90 msgr2=0x7f36f414ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:37.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f1d9b640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36f40c5c90 0x7f36f414ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f1d9b640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f40c1490 con 0x7f36f40c2cf0 2024-09-17T05:31:37.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f2d9d640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36f40c5c90 0x7f36f414ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:31:37.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.062+0000 7f36f259c640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 0x7f36f415c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:31:37.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.063+0000 7f36f1d9b640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f40c2cf0 0x7f36f40c0a60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f36f4072b70 tx=0x7f36e400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:37.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.068+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4015020 con 0x7f36f40c2cf0 2024-09-17T05:31:37.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.068+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f41f54e0 con 0x7f36f40c2cf0 2024-09-17T05:31:37.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.068+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f40c0260 con 0x7f36f40c2cf0 2024-09-17T05:31:37.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.070+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36e4011920 con 0x7f36f40c2cf0 2024-09-17T05:31:37.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.071+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4015020 con 0x7f36f40c2cf0 2024-09-17T05:31:37.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.074+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f36e4018020 con 0x7f36f40c2cf0 2024-09-17T05:31:37.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.076+0000 7f36e2ffd640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f36d4078fc0 0x7f36d407b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:37.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.076+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d8005290 con 0x7f36f40c2cf0 2024-09-17T05:31:37.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.077+0000 7f36f259c640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f36d4078fc0 0x7f36d407b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:37.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.077+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f36e4011630 con 0x7f36f40c2cf0 2024-09-17T05:31:37.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.081+0000 7f36f259c640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f36d4078fc0 0x7f36d407b4d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001050 tx=0x7f36e8006040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:37.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.090+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f36e4067e20 con 0x7f36f40c2cf0 2024-09-17T05:31:37.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.586+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36d8005850 con 0x7f36f40c2cf0 2024-09-17T05:31:37.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.587+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1522454267 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7f36e4067e20 con 0x7f36f40c2cf0 2024-09-17T05:31:37.591 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:37.591 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f36d4078fc0 msgr2=0x7f36d407b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f36d4078fc0 0x7f36d407b4d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001050 tx=0x7f36e8006040 comp rx=0 tx=0).stop 2024-09-17T05:31:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f40c2cf0 msgr2=0x7f36f40c0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f40c2cf0 0x7f36f40c0a60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f36f4072b70 tx=0x7f36e400b7b0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 shutdown_connections 2024-09-17T05:31:37.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f36d4078fc0 0x7f36d407b4d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36f40c5c90 0x7f36f414ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.592+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f40c2cf0 0x7f36f40c0a60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.593+0000 7f36f359e640 1 --2- 172.21.15.117:0/1522454267 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f36f415f2b0 0x7f36f415c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:37.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.593+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 >> 172.21.15.117:0/1522454267 conn(0x7f36f40c8d10 msgr2=0x7f36f4150d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:37.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.594+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 shutdown_connections 2024-09-17T05:31:37.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:37.594+0000 7f36f359e640 1 -- 172.21.15.117:0/1522454267 wait complete. 2024-09-17T05:31:37.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:39.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:39.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.826+0000 7fc855ffb640 1 -- 172.21.15.117:0/2001011916 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848005280 con 0x7fc85004c5b0 2024-09-17T05:31:39.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.826+0000 7fc857fff640 1 -- 172.21.15.117:0/2001011916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc85004c5b0 msgr2=0x7fc8500dac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:39.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.826+0000 7fc857fff640 1 --2- 172.21.15.117:0/2001011916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc85004c5b0 0x7fc8500dac50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc8500501c0 tx=0x7fc84802b250 comp rx=0 tx=0).stop 2024-09-17T05:31:39.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.827+0000 7fc857fff640 1 -- 172.21.15.117:0/2001011916 shutdown_connections 2024-09-17T05:31:39.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.827+0000 7fc857fff640 1 --2- 172.21.15.117:0/2001011916 >> v2:172.21.15.117:3300/0 conn(0x7fc850045c00 0x7fc850045100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:39.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.827+0000 7fc857fff640 1 --2- 172.21.15.117:0/2001011916 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8500d5070 0x7fc8500469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:39.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.827+0000 7fc857fff640 1 --2- 172.21.15.117:0/2001011916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc85004c5b0 0x7fc8500dac50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:39.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.827+0000 7fc857fff640 1 -- 172.21.15.117:0/2001011916 >> 172.21.15.117:0/2001011916 conn(0x7fc850050280 msgr2=0x7fc850050690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:39.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.828+0000 7fc857fff640 1 -- 172.21.15.117:0/2001011916 shutdown_connections 2024-09-17T05:31:39.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.828+0000 7fc857fff640 1 -- 172.21.15.117:0/2001011916 wait complete. 2024-09-17T05:31:39.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.833+0000 7fc857fff640 1 Processor -- start 2024-09-17T05:31:39.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.833+0000 7fc857fff640 1 -- start start 2024-09-17T05:31:39.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.835+0000 7fc857fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc850045c00 0x7fc8501763f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:39.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.835+0000 7fc857fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 0x7fc8500d9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc856ffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc850045c00 0x7fc8501763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc856ffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc850045c00 0x7fc8501763f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:57604/0 (socket says 172.21.15.117:57604) 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc8567fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 0x7fc8500d9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc8567fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 0x7fc8500d9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37704/0 (socket says 172.21.15.117:37704) 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc856ffd640 1 -- 172.21.15.117:0/1513617059 learned_addr learned my addr 172.21.15.117:0/1513617059 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8500d5070 0x7fc8500c2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc8500461f0 con 0x7fc85004c5b0 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc8500dbb80 con 0x7fc8500d5070 2024-09-17T05:31:39.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.836+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc850006050 con 0x7fc850045c00 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8577fe640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8500d5070 0x7fc8500c2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8567fc640 1 -- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc850045c00 msgr2=0x7fc8501763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8567fc640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc850045c00 0x7fc8501763f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8567fc640 1 -- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8500d5070 msgr2=0x7fc8500c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8567fc640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8500d5070 0x7fc8500c2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8567fc640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8500da1c0 con 0x7fc85004c5b0 2024-09-17T05:31:39.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.837+0000 7fc8567fc640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 0x7fc8500d9c80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00d650 tx=0x7fc84c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:39.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.841+0000 7fc8577fe640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8500d5070 0x7fc8500c2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:31:39.842 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.842+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c017070 con 0x7fc85004c5b0 2024-09-17T05:31:39.842 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.842+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc84c005310 con 0x7fc85004c5b0 2024-09-17T05:31:39.842 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.842+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c009820 con 0x7fc85004c5b0 2024-09-17T05:31:39.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.843+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8500cc210 con 0x7fc85004c5b0 2024-09-17T05:31:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.846+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8500d9480 con 0x7fc85004c5b0 2024-09-17T05:31:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.846+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc84c0099c0 con 0x7fc85004c5b0 2024-09-17T05:31:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.848+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8500d85e0 con 0x7fc85004c5b0 2024-09-17T05:31:39.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.850+0000 7fc83f7fe640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc844078bc0 0x7fc84407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:39.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.850+0000 7fc856ffd640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc844078bc0 0x7fc84407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:39.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.851+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fc84c09a4a0 con 0x7fc85004c5b0 2024-09-17T05:31:39.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.854+0000 7fc856ffd640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc844078bc0 0x7fc84407b0d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc848000c00 tx=0x7fc84802d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:39.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:39.857+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc84c061ed0 con 0x7fc85004c5b0 2024-09-17T05:31:40.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.259+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc850033cd0 con 0x7fc85004c5b0 2024-09-17T05:31:40.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.260+0000 7fc83f7fe640 1 -- 172.21.15.117:0/1513617059 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7fc84c066900 con 0x7fc85004c5b0 2024-09-17T05:31:40.261 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:40.261 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:40.261 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:40.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc844078bc0 msgr2=0x7fc84407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:40.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc844078bc0 0x7fc84407b0d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc848000c00 tx=0x7fc84802d040 comp rx=0 tx=0).stop 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 msgr2=0x7fc8500d9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 0x7fc8500d9c80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00d650 tx=0x7fc84c00db10 comp rx=0 tx=0).stop 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 shutdown_connections 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc844078bc0 0x7fc84407b0d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8500d5070 0x7fc8500c2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc85004c5b0 0x7fc8500d9c80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 --2- 172.21.15.117:0/1513617059 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc850045c00 0x7fc8501763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 >> 172.21.15.117:0/1513617059 conn(0x7fc850050280 msgr2=0x7fc850005870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 shutdown_connections 2024-09-17T05:31:40.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:40.263+0000 7fc857fff640 1 -- 172.21.15.117:0/1513617059 wait complete. 2024-09-17T05:31:40.279 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:42.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:42.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.458+0000 7f7ae915d640 1 -- 172.21.15.117:0/3041143284 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4031a10 con 0x7f7ae4073930 2024-09-17T05:31:42.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 -- 172.21.15.117:0/3041143284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae4073930 msgr2=0x7f7ae415e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:42.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/3041143284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae4073930 0x7f7ae415e9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40c7340 tx=0x7f7ad402a640 comp rx=0 tx=0).stop 2024-09-17T05:31:42.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 -- 172.21.15.117:0/3041143284 shutdown_connections 2024-09-17T05:31:42.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/3041143284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae4073930 0x7f7ae415e9f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 -- 172.21.15.117:0/3041143284 >> 172.21.15.117:0/3041143284 conn(0x7f7ae40c9a20 msgr2=0x7f7ae40c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:42.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 -- 172.21.15.117:0/3041143284 shutdown_connections 2024-09-17T05:31:42.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.460+0000 7f7aec3c3640 1 -- 172.21.15.117:0/3041143284 wait complete. 2024-09-17T05:31:42.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.462+0000 7f7aec3c3640 1 Processor -- start 2024-09-17T05:31:42.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.463+0000 7f7aec3c3640 1 -- start start 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7aec3c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ae4073930 0x7f7ae41e70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7aec3c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 0x7f7ae41e4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7aec3c3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae415ef30 0x7f7ae4154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7ae995e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 0x7f7ae41e4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7ae995e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 0x7f7ae41e4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37710/0 (socket says 172.21.15.117:37710) 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7ae995e640 1 -- 172.21.15.117:0/1891541336 learned_addr learned my addr 172.21.15.117:0/1891541336 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:42.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.465+0000 7f7aea15f640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ae4073930 0x7f7ae41e70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:42.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4073550 con 0x7f7ae40c8420 2024-09-17T05:31:42.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7aea960640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae415ef30 0x7f7ae4154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:42.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae415f6a0 con 0x7f7ae415ef30 2024-09-17T05:31:42.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae40736d0 con 0x7f7ae4073930 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7ae995e640 1 -- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ae4073930 msgr2=0x7f7ae41e70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7ae995e640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ae4073930 0x7f7ae41e70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7ae995e640 1 -- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae415ef30 msgr2=0x7f7ae4154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7ae995e640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae415ef30 0x7f7ae4154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.466+0000 7f7ae995e640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae41e5820 con 0x7f7ae40c8420 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.467+0000 7f7ae995e640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 0x7f7ae41e4fb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c3d0 tx=0x7f7ae000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.467+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0011790 con 0x7f7ae40c8420 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.467+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ae0017070 con 0x7f7ae40c8420 2024-09-17T05:31:42.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.467+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0011cc0 con 0x7f7ae40c8420 2024-09-17T05:31:42.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.467+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae41e54f0 con 0x7f7ae40c8420 2024-09-17T05:31:42.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.467+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae41f2590 con 0x7f7ae40c8420 2024-09-17T05:31:42.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.468+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae406d220 con 0x7f7ae40c8420 2024-09-17T05:31:42.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.469+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7ae001c460 con 0x7f7ae40c8420 2024-09-17T05:31:42.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.470+0000 7f7adaffd640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7ac8078bc0 0x7f7ac807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:42.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.470+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7f7ae009ad80 con 0x7f7ae40c8420 2024-09-17T05:31:42.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.470+0000 7f7aea15f640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7ac8078bc0 0x7f7ac807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:42.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.471+0000 7f7aea15f640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7ac8078bc0 0x7f7ac807b0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4034000 tx=0x7f7ad4004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:42.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.477+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7ae0062830 con 0x7f7ae40c8420 2024-09-17T05:31:42.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.772+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ae406cf40 con 0x7f7ae40c8420 2024-09-17T05:31:42.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.772+0000 7f7adaffd640 1 -- 172.21.15.117:0/1891541336 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7f7ae0067260 con 0x7f7ae40c8420 2024-09-17T05:31:42.773 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:42.773 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:42.774 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:42.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7ac8078bc0 msgr2=0x7f7ac807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:42.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7ac8078bc0 0x7f7ac807b0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4034000 tx=0x7f7ad4004730 comp rx=0 tx=0).stop 2024-09-17T05:31:42.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 msgr2=0x7f7ae41e4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 0x7f7ae41e4fb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c3d0 tx=0x7f7ae000c890 comp rx=0 tx=0).stop 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 shutdown_connections 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7ac8078bc0 0x7f7ac807b0d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ae415ef30 0x7f7ae4154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae40c8420 0x7f7ae41e4fb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 --2- 172.21.15.117:0/1891541336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ae4073930 0x7f7ae41e70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 >> 172.21.15.117:0/1891541336 conn(0x7f7ae40c9a20 msgr2=0x7f7ae415dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 shutdown_connections 2024-09-17T05:31:42.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:42.775+0000 7f7aec3c3640 1 -- 172.21.15.117:0/1891541336 wait complete. 2024-09-17T05:31:42.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:44.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:45.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.015+0000 7fb0da7fc640 1 -- 172.21.15.117:0/2402756104 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4006510 con 0x7fb0c8003a50 2024-09-17T05:31:45.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.015+0000 7fb0dbfff640 1 -- 172.21.15.117:0/2402756104 reap_dead start 2024-09-17T05:31:45.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.016+0000 7fb0e204d640 1 -- 172.21.15.117:0/2402756104 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0c8003a50 msgr2=0x7fb0c8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:45.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.016+0000 7fb0e204d640 1 --2- 172.21.15.117:0/2402756104 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0c8003a50 0x7fb0c8003ec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400c610 tx=0x7fb0c8005830 comp rx=0 tx=0).stop 2024-09-17T05:31:45.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.020+0000 7fb0e204d640 1 -- 172.21.15.117:0/2402756104 shutdown_connections 2024-09-17T05:31:45.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.020+0000 7fb0e204d640 1 --2- 172.21.15.117:0/2402756104 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0c8003a50 0x7fb0c8003ec0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.020+0000 7fb0e204d640 1 -- 172.21.15.117:0/2402756104 >> 172.21.15.117:0/2402756104 conn(0x7fb0dc06e020 msgr2=0x7fb0dc06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.022+0000 7fb0e204d640 1 -- 172.21.15.117:0/2402756104 shutdown_connections 2024-09-17T05:31:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.022+0000 7fb0e204d640 1 -- 172.21.15.117:0/2402756104 wait complete. 2024-09-17T05:31:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.025+0000 7fb0e204d640 1 Processor -- start 2024-09-17T05:31:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.025+0000 7fb0e204d640 1 -- start start 2024-09-17T05:31:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.027+0000 7fb0e204d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0c8003a50 0x7fb0dc1f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0e204d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0dc150ab0 0x7fb0dc159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0e204d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 0x7fb0dc14c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0e204d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc0c7120 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0e204d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc15bc30 con 0x7fb0c8003a50 2024-09-17T05:31:45.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0e204d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb0dc0c72a0 con 0x7fb0dc150ab0 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0db7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0c8003a50 0x7fb0dc1f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0dbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 0x7fb0dc14c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0dbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 0x7fb0dc14c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34138/0 (socket says 172.21.15.117:34138) 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.028+0000 7fb0dbfff640 1 -- 172.21.15.117:0/1160775600 learned_addr learned my addr 172.21.15.117:0/1160775600 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.029+0000 7fb0daffd640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0dc150ab0 0x7fb0dc159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.029+0000 7fb0dbfff640 1 -- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0dc150ab0 msgr2=0x7fb0dc159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.029+0000 7fb0dbfff640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0dc150ab0 0x7fb0dc159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.029+0000 7fb0dbfff640 1 -- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0c8003a50 msgr2=0x7fb0dc1f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.029+0000 7fb0dbfff640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0c8003a50 0x7fb0dc1f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.029+0000 7fb0dbfff640 1 -- 172.21.15.117:0/1160775600 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0dc158080 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.030+0000 7fb0dbfff640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 0x7fb0dc14c950 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0c7b40 tx=0x7fb0c400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.030+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4004220 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.030+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0dc1f0b50 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.031+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0dc1582f0 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.031+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0c4005020 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.031+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4005470 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.036+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb0c40047a0 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.036+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c0005290 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.036+0000 7fb0d3fff640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb0bc078be0 0x7fb0bc07b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.036+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6676+0+0 (secure 0 0 0) 0x7fb0c409b2e0 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.039+0000 7fb0db7fe640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb0bc078be0 0x7fb0bc07b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.043+0000 7fb0db7fe640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb0bc078be0 0x7fb0bc07b0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4001eb0 tx=0x7fb0d4001bc0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:45.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.047+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb0c4063450 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.526+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0c0005580 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.528+0000 7fb0d3fff640 1 -- 172.21.15.117:0/1160775600 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6927 (secure 0 0 0) 0x7fb0c4062d50 con 0x7fb0dc0c7e20 2024-09-17T05:31:45.528 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:45.529 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":20,"btime":"2024-09-17T05:29:32:008305+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5082,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:29:31.001980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":4419},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4419":{"gid":4419,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":36,"addr":"172.21.15.117:6835/235108685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":235108685},{"type":"v1","addr":"172.21.15.117:6835","nonce":235108685}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434,4419]},"id":1}]} 2024-09-17T05:31:45.529 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 20 2024-09-17T05:31:45.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.532+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb0bc078be0 msgr2=0x7fb0bc07b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:45.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.532+0000 7fb0e204d640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb0bc078be0 0x7fb0bc07b0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4001eb0 tx=0x7fb0d4001bc0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.532+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 msgr2=0x7fb0dc14c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:45.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.532+0000 7fb0e204d640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 0x7fb0dc14c950 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0c7b40 tx=0x7fb0c400e6c0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 shutdown_connections 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb0bc078be0 0x7fb0bc07b0f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0dc0c7e20 0x7fb0dc14c950 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0dc150ab0 0x7fb0dc159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 --2- 172.21.15.117:0/1160775600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0c8003a50 0x7fb0dc1f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 >> 172.21.15.117:0/1160775600 conn(0x7fb0dc06e020 msgr2=0x7fb0dc068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 shutdown_connections 2024-09-17T05:31:45.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:45.533+0000 7fb0e204d640 1 -- 172.21.15.117:0/1160775600 wait complete. 2024-09-17T05:31:45.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:31:47.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 -- 172.21.15.117:0/3988622073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c6f50 msgr2=0x7fba98073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 --2- 172.21.15.117:0/3988622073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c6f50 0x7fba98073f60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba980cde70 tx=0x7fba8802a4d0 comp rx=0 tx=0).stop 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 -- 172.21.15.117:0/3988622073 shutdown_connections 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 --2- 172.21.15.117:0/3988622073 >> v2:172.21.15.117:3300/0 conn(0x7fba9815b130 0x7fba980b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 --2- 172.21.15.117:0/3988622073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fba980c8390 0x7fba9815abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 --2- 172.21.15.117:0/3988622073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c6f50 0x7fba98073f60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 -- 172.21.15.117:0/3988622073 >> 172.21.15.117:0/3988622073 conn(0x7fba980c9a20 msgr2=0x7fba980c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 -- 172.21.15.117:0/3988622073 shutdown_connections 2024-09-17T05:31:47.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.734+0000 7fba9e320640 1 -- 172.21.15.117:0/3988622073 wait complete. 2024-09-17T05:31:47.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.735+0000 7fba9e320640 1 Processor -- start 2024-09-17T05:31:47.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.735+0000 7fba9e320640 1 -- start start 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9e320640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fba980c6f50 0x7fba98147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9e320640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c8390 0x7fba980c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9e320640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 0x7fba98156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9e320640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fba98157dd0 con 0x7fba9815b130 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9e320640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fba98157af0 con 0x7fba980c8390 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9e320640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fba980c87a0 con 0x7fba980c6f50 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 0x7fba98156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 0x7fba98156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34156/0 (socket says 172.21.15.117:34156) 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 -- 172.21.15.117:0/1060215232 learned_addr learned my addr 172.21.15.117:0/1060215232 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba8ffff640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c8390 0x7fba980c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:47.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba97fff640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fba980c6f50 0x7fba98147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 -- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fba980c6f50 msgr2=0x7fba98147f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fba980c6f50 0x7fba98147f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 -- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c8390 msgr2=0x7fba980c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c8390 0x7fba980c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 -- 172.21.15.117:0/1060215232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9814d110 con 0x7fba9815b130 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.737+0000 7fba9c8bd640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 0x7fba98156740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba9000ec20 tx=0x7fba9000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.738+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90003b70 con 0x7fba9815b130 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.738+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba90003e00 con 0x7fba9815b130 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.738+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90004280 con 0x7fba9815b130 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.738+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba98148460 con 0x7fba9815b130 2024-09-17T05:31:47.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.738+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba9814ce30 con 0x7fba9815b130 2024-09-17T05:31:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.739+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba980c94d0 con 0x7fba9815b130 2024-09-17T05:31:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.740+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fba90004420 con 0x7fba9815b130 2024-09-17T05:31:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.740+0000 7fba957fa640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba78078b90 0x7fba7807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.740+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fba9009bdd0 con 0x7fba9815b130 2024-09-17T05:31:47.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.740+0000 7fba97fff640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba78078b90 0x7fba7807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:47.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.741+0000 7fba97fff640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba78078b90 0x7fba7807b0a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fba88034000 tx=0x7fba88004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:47.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:47.746+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fba90063df0 con 0x7fba9815b130 2024-09-17T05:31:48.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.051+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba9814d8a0 con 0x7fba9815b130 2024-09-17T05:31:48.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.052+0000 7fba957fa640 1 -- 172.21.15.117:0/1060215232 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7fba9000ee60 con 0x7fba9815b130 2024-09-17T05:31:48.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:48.053 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:31:48.053 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba78078b90 msgr2=0x7fba7807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba78078b90 0x7fba7807b0a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fba88034000 tx=0x7fba88004730 comp rx=0 tx=0).stop 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 msgr2=0x7fba98156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 0x7fba98156740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba9000ec20 tx=0x7fba9000c6a0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 shutdown_connections 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fba78078b90 0x7fba7807b0a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba9815b130 0x7fba98156740 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980c8390 0x7fba980c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 --2- 172.21.15.117:0/1060215232 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fba980c6f50 0x7fba98147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 >> 172.21.15.117:0/1060215232 conn(0x7fba980c9a20 msgr2=0x7fba980c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:48.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 shutdown_connections 2024-09-17T05:31:48.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.055+0000 7fba9e320640 1 -- 172.21.15.117:0/1060215232 wait complete. 2024-09-17T05:31:48.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-09-17T05:31:48.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-09-17T05:31:48.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 -- 172.21.15.117:0/3823809970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 msgr2=0x7fe9a4162a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/3823809970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 0x7fe9a4162a40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40ce040 tx=0x7fe99c02a820 comp rx=0 tx=0).stop 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 -- 172.21.15.117:0/3823809970 shutdown_connections 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/3823809970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 0x7fe9a4162a40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/3823809970 >> v2:172.21.15.117:3300/0 conn(0x7fe9a4152210 0x7fe9a40c4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 -- 172.21.15.117:0/3823809970 >> 172.21.15.117:0/3823809970 conn(0x7fe9a406e590 msgr2=0x7fe9a40cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:48.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 -- 172.21.15.117:0/3823809970 shutdown_connections 2024-09-17T05:31:48.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.233+0000 7fe9a94cc640 1 -- 172.21.15.117:0/3823809970 wait complete. 2024-09-17T05:31:48.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.236+0000 7fe9a94cc640 1 Processor -- start 2024-09-17T05:31:48.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.236+0000 7fe9a94cc640 1 -- start start 2024-09-17T05:31:48.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.237+0000 7fe9a94cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 0x7fe9a40ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:48.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.237+0000 7fe9a94cc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9a40c67a0 0x7fe9a41535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:48.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.237+0000 7fe9a94cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 0x7fe9a4153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:48.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.237+0000 7fe9a94cc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a40ce0d0 con 0x7fe9a4152210 2024-09-17T05:31:48.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.237+0000 7fe9a94cc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a40c58a0 con 0x7fe9a40c5490 2024-09-17T05:31:48.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a94cc640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a40c7ff0 con 0x7fe9a40c67a0 2024-09-17T05:31:48.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 0x7fe9a4153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:48.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 0x7fe9a4153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34170/0 (socket says 172.21.15.117:34170) 2024-09-17T05:31:48.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 -- 172.21.15.117:0/2850068298 learned_addr learned my addr 172.21.15.117:0/2850068298 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:48.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a27fc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9a40c67a0 0x7fe9a41535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:48.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 -- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9a40c67a0 msgr2=0x7fe9a41535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9a40c67a0 0x7fe9a41535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 -- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 msgr2=0x7fe9a40ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 0x7fe9a40ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.238+0000 7fe9a37fe640 1 -- 172.21.15.117:0/2850068298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a41529b0 con 0x7fe9a4152210 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.239+0000 7fe9a37fe640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 0x7fe9a4153af0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02ae10 tx=0x7fe99c032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.240+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c037040 con 0x7fe9a4152210 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.240+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe99c032e50 con 0x7fe9a4152210 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.240+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c003e60 con 0x7fe9a4152210 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.240+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a4152c90 con 0x7fe9a4152210 2024-09-17T05:31:48.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.240+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a4072160 con 0x7fe9a4152210 2024-09-17T05:31:48.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.242+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a4158660 con 0x7fe9a4152210 2024-09-17T05:31:48.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.243+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe99c044020 con 0x7fe9a4152210 2024-09-17T05:31:48.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.244+0000 7fe98b7fe640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe990078be0 0x7fe99007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:48.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.244+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fe99c0bf890 con 0x7fe9a4152210 2024-09-17T05:31:48.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.244+0000 7fe9a2ffd640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe990078be0 0x7fe99007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:48.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.246+0000 7fe9a2ffd640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe990078be0 0x7fe99007b0f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe994007fd0 tx=0x7fe9940078e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:48.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.249+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe99c087ab0 con 0x7fe9a4152210 2024-09-17T05:31:48.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.551+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9a40c8d00 con 0x7fe9a4152210 2024-09-17T05:31:48.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.552+0000 7fe98b7fe640 1 -- 172.21.15.117:0/2850068298 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7fe99c0873b0 con 0x7fe9a4152210 2024-09-17T05:31:48.552 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:48.553 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:31:48.553 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe990078be0 msgr2=0x7fe99007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe990078be0 0x7fe99007b0f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe994007fd0 tx=0x7fe9940078e0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 msgr2=0x7fe9a4153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 0x7fe9a4153af0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02ae10 tx=0x7fe99c032c60 comp rx=0 tx=0).stop 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 shutdown_connections 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe990078be0 0x7fe99007b0f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9a4152210 0x7fe9a4153af0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9a40c67a0 0x7fe9a41535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 --2- 172.21.15.117:0/2850068298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a40c5490 0x7fe9a40ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:48.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 >> 172.21.15.117:0/2850068298 conn(0x7fe9a406e590 msgr2=0x7fe9a4151010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:48.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 shutdown_connections 2024-09-17T05:31:48.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:48.554+0000 7fe9a94cc640 1 -- 172.21.15.117:0/2850068298 wait complete. 2024-09-17T05:31:48.569 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:31:45:857416+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4434": { "addr": "172.21.15.119:6835/2224210538", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2224210538, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2224210538, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4434, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:31:45.857410+0000", "qdb_cluster": [ 4434, 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 4434, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:31:50.570 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:50.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 -- 172.21.15.117:0/2372671549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25940c92c0 msgr2=0x7f25940c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 --2- 172.21.15.117:0/2372671549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25940c92c0 0x7f25940c96d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2588005fd0 tx=0x7f258802a5d0 comp rx=0 tx=0).stop 2024-09-17T05:31:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 -- 172.21.15.117:0/2372671549 shutdown_connections 2024-09-17T05:31:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 --2- 172.21.15.117:0/2372671549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25940c92c0 0x7f25940c96d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 --2- 172.21.15.117:0/2372671549 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f25941638c0 0x7f25941601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:50.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 -- 172.21.15.117:0/2372671549 >> 172.21.15.117:0/2372671549 conn(0x7f259406d190 msgr2=0x7f259406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:50.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 -- 172.21.15.117:0/2372671549 shutdown_connections 2024-09-17T05:31:50.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.782+0000 7f259c09b640 1 -- 172.21.15.117:0/2372671549 wait complete. 2024-09-17T05:31:50.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.785+0000 7f259c09b640 1 Processor -- start 2024-09-17T05:31:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.785+0000 7f259c09b640 1 -- start start 2024-09-17T05:31:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.787+0000 7f259c09b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 0x7f25942072e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:50.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.788+0000 7f259c09b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25940c92c0 0x7f2594200ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:50.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.790+0000 7f259c09b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25941638c0 0x7f25940ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:50.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.790+0000 7f259c09b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f25940c8540 con 0x7f25940c92c0 2024-09-17T05:31:50.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.790+0000 7f259c09b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2594163cd0 con 0x7f25941638c0 2024-09-17T05:31:50.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.790+0000 7f259c09b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2594162de0 con 0x7f2594073460 2024-09-17T05:31:50.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.793+0000 7f2599e37640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 0x7f25942072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:50.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.793+0000 7f2599e37640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 0x7f25942072e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:34258/0 (socket says 172.21.15.117:34258) 2024-09-17T05:31:50.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.793+0000 7f2599e37640 1 -- 172.21.15.117:0/3627272423 learned_addr learned my addr 172.21.15.117:0/3627272423 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:50.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.798+0000 7f2599e37640 1 -- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25941638c0 msgr2=0x7f25940ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.798+0000 7f2599e37640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25941638c0 0x7f25940ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.798+0000 7f2599e37640 1 -- 172.21.15.117:0/3627272423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25940c92c0 msgr2=0x7f2594200ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.801+0000 7f2599636640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25940c92c0 0x7f2594200ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.801+0000 7f2599e37640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25940c92c0 0x7f2594200ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.801+0000 7f2599e37640 1 -- 172.21.15.117:0/3627272423 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2594207820 con 0x7f2594073460 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.801+0000 7f2599636640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25940c92c0 0x7f2594200ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.802+0000 7f2599e37640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 0x7f25942072e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f258c00c7b0 tx=0x7f258c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.803+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c003b70 con 0x7f2594073460 2024-09-17T05:31:50.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.803+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f258c003d10 con 0x7f2594073460 2024-09-17T05:31:50.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.803+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c00aa50 con 0x7f2594073460 2024-09-17T05:31:50.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.804+0000 7f259c09b640 1 -- 172.21.15.117:0/3627272423 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2594201220 con 0x7f2594073460 2024-09-17T05:31:50.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.808+0000 7f259c09b640 1 -- 172.21.15.117:0/3627272423 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2594206b00 con 0x7f2594073460 2024-09-17T05:31:50.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.810+0000 7f259c09b640 1 -- 172.21.15.117:0/3627272423 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25940c39c0 con 0x7f2594073460 2024-09-17T05:31:50.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.816+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f258c00abf0 con 0x7f2594073460 2024-09-17T05:31:50.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.817+0000 7f25867fc640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2578078bc0 0x7f257807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:50.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.817+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f258c09b080 con 0x7f2594073460 2024-09-17T05:31:50.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.818+0000 7f2599636640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2578078bc0 0x7f257807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:50.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.820+0000 7f2599636640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2578078bc0 0x7f257807b0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f259415d190 tx=0x7f2590008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:50.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:50.826+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f25940c39c0 con 0x7f2594073460 2024-09-17T05:31:51.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.271+0000 7f259c09b640 1 -- 172.21.15.117:0/3627272423 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25941630c0 con 0x7f2594073460 2024-09-17T05:31:51.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.274+0000 7f25867fc640 1 -- 172.21.15.117:0/3627272423 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7f258c020060 con 0x7f2594073460 2024-09-17T05:31:51.275 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:51.275 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:31:51.275 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:31:51.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.279+0000 7f2577fff640 1 -- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2578078bc0 msgr2=0x7f257807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:51.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.279+0000 7f2577fff640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2578078bc0 0x7f257807b0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f259415d190 tx=0x7f2590008040 comp rx=0 tx=0).stop 2024-09-17T05:31:51.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.279+0000 7f2577fff640 1 -- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 msgr2=0x7f25942072e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:51.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.279+0000 7f2577fff640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 0x7f25942072e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f258c00c7b0 tx=0x7f258c00cc70 comp rx=0 tx=0).stop 2024-09-17T05:31:51.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 -- 172.21.15.117:0/3627272423 shutdown_connections 2024-09-17T05:31:51.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2578078bc0 0x7f257807b0d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:51.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25941638c0 0x7f25940ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:51.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25940c92c0 0x7f2594200ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:51.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 --2- 172.21.15.117:0/3627272423 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2594073460 0x7f25942072e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:51.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 -- 172.21.15.117:0/3627272423 >> 172.21.15.117:0/3627272423 conn(0x7f259406d190 msgr2=0x7f259415ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:51.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 -- 172.21.15.117:0/3627272423 shutdown_connections 2024-09-17T05:31:51.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:51.280+0000 7f2577fff640 1 -- 172.21.15.117:0/3627272423 wait complete. 2024-09-17T05:31:51.305 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:31:53.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:53.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.506+0000 7f0058e64640 1 -- 172.21.15.117:0/3636917207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054073ae0 msgr2=0x7f00541518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:53.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.506+0000 7f005159a640 1 -- 172.21.15.117:0/3636917207 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c035020 con 0x7f0054073ae0 2024-09-17T05:31:53.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.506+0000 7f0058e64640 1 --2- 172.21.15.117:0/3636917207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054073ae0 0x7f00541518f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f004c005fd0 tx=0x7f004c02a5f0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.507+0000 7f0058e64640 1 -- 172.21.15.117:0/3636917207 shutdown_connections 2024-09-17T05:31:53.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.507+0000 7f0058e64640 1 --2- 172.21.15.117:0/3636917207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054073ae0 0x7f00541518f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.507+0000 7f0058e64640 1 --2- 172.21.15.117:0/3636917207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0054074030 0x7f00540c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.507+0000 7f0058e64640 1 -- 172.21.15.117:0/3636917207 >> 172.21.15.117:0/3636917207 conn(0x7f005406d190 msgr2=0x7f005406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:53.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.507+0000 7f0058e64640 1 -- 172.21.15.117:0/3636917207 shutdown_connections 2024-09-17T05:31:53.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.508+0000 7f0058e64640 1 -- 172.21.15.117:0/3636917207 wait complete. 2024-09-17T05:31:53.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.513+0000 7f0058e64640 1 Processor -- start 2024-09-17T05:31:53.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.514+0000 7f0058e64640 1 -- start start 2024-09-17T05:31:53.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.516+0000 7f0058e64640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0054073ae0 0x7f005414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:53.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.516+0000 7f0058e64640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054074030 0x7f00540cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:53.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f005259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0054073ae0 0x7f005414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0058e64640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 0x7f00540cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0058e64640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f00540c9300 con 0x7f0054074030 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0058e64640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0054073270 con 0x7f0054073ae0 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 0x7f00540cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0052d9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 0x7f00540cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:34278/0 (socket says 172.21.15.117:34278) 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0052d9d640 1 -- 172.21.15.117:0/1412648088 learned_addr learned my addr 172.21.15.117:0/1412648088 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f00540c9480 con 0x7f00540c81b0 2024-09-17T05:31:53.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.517+0000 7f005259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0054073ae0 0x7f005414ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44674/0 (socket says 172.21.15.117:44674) 2024-09-17T05:31:53.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.519+0000 7f0052d9d640 1 -- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0054073ae0 msgr2=0x7f005414ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:53.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.519+0000 7f0052d9d640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0054073ae0 0x7f005414ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.519+0000 7f0052d9d640 1 -- 172.21.15.117:0/1412648088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054074030 msgr2=0x7f00540cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:53.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.519+0000 7f0052d9d640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054074030 0x7f00540cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.519+0000 7f0052d9d640 1 -- 172.21.15.117:0/1412648088 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005413aed0 con 0x7f00540c81b0 2024-09-17T05:31:53.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.520+0000 7f0052d9d640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 0x7f00540cca50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f004c0099f0 tx=0x7f004c003910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:53.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.522+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c03e030 con 0x7f00540c81b0 2024-09-17T05:31:53.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.522+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f004c003e00 con 0x7f00540c81b0 2024-09-17T05:31:53.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.522+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c035040 con 0x7f00540c81b0 2024-09-17T05:31:53.524 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.523+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005413b1b0 con 0x7f00540c81b0 2024-09-17T05:31:53.524 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.524+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f005413b770 con 0x7f00540c81b0 2024-09-17T05:31:53.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.529+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00541518f0 con 0x7f00540c81b0 2024-09-17T05:31:53.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.533+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f004c041030 con 0x7f00540c81b0 2024-09-17T05:31:53.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.534+0000 7f004affd640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0034078c10 0x7f003407b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:53.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.534+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f004c0bbe20 con 0x7f00540c81b0 2024-09-17T05:31:53.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.535+0000 7f005259c640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0034078c10 0x7f003407b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:53.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.538+0000 7f005259c640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0034078c10 0x7f003407b120 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:53.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.540+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f004c083960 con 0x7f00540c81b0 2024-09-17T05:31:53.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.954+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0054142580 con 0x7f00540c81b0 2024-09-17T05:31:53.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.956+0000 7f004affd640 1 -- 172.21.15.117:0/1412648088 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7f004c088390 con 0x7f00540c81b0 2024-09-17T05:31:53.959 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:53.959 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:31:53.959 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:31:53.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.960+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0034078c10 msgr2=0x7f003407b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:53.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.960+0000 7f0058e64640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0034078c10 0x7f003407b120 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c009060 comp rx=0 tx=0).stop 2024-09-17T05:31:53.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.960+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 msgr2=0x7f00540cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:53.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.960+0000 7f0058e64640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 0x7f00540cca50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f004c0099f0 tx=0x7f004c003910 comp rx=0 tx=0).stop 2024-09-17T05:31:53.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.961+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 shutdown_connections 2024-09-17T05:31:53.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.961+0000 7f0058e64640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0034078c10 0x7f003407b120 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f003c006fd0 tx=0x7f003c009060 comp rx=0 tx=0).stop 2024-09-17T05:31:53.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.961+0000 7f0058e64640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f00540c81b0 0x7f00540cca50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.961+0000 7f0058e64640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0054074030 0x7f00540cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.961+0000 7f0058e64640 1 --2- 172.21.15.117:0/1412648088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0054073ae0 0x7f005414ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:53.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.961+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 >> 172.21.15.117:0/1412648088 conn(0x7f005406d190 msgr2=0x7f005414c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:53.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.962+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 shutdown_connections 2024-09-17T05:31:53.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:53.962+0000 7f0058e64640 1 -- 172.21.15.117:0/1412648088 wait complete. 2024-09-17T05:31:53.981 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:31:53.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:31:53.989 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:31:55.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:56.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.122+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1245701619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e415ef80 msgr2=0x7fa3e41554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:56.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.122+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1245701619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e415ef80 0x7fa3e41554e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009830 tx=0x7fa3d002a5b0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.124+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1245701619 shutdown_connections 2024-09-17T05:31:56.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.124+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1245701619 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa3e40c5ad0 0x7fa3e40cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.124+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1245701619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e415ef80 0x7fa3e41554e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.124+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1245701619 >> 172.21.15.117:0/1245701619 conn(0x7fa3e406e020 msgr2=0x7fa3e406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:56.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.124+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1245701619 shutdown_connections 2024-09-17T05:31:56.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.124+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1245701619 wait complete. 2024-09-17T05:31:56.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.128+0000 7fa3eabdf640 1 Processor -- start 2024-09-17T05:31:56.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.128+0000 7fa3eabdf640 1 -- start start 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.130+0000 7fa3eabdf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 0x7fa3e41f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.130+0000 7fa3eabdf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e40c61d0 0x7fa3e414f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.130+0000 7fa3eabdf640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa3e415ef80 0x7fa3e40ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.130+0000 7fa3eabdf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e40c7120 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.130+0000 7fa3eabdf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e415bc30 con 0x7fa3e40c61d0 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.130+0000 7fa3eabdf640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa3e40c72a0 con 0x7fa3e415ef80 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 0x7fa3e41f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 0x7fa3e41f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:33186/0 (socket says 172.21.15.117:33186) 2024-09-17T05:31:56.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 -- 172.21.15.117:0/1443450206 learned_addr learned my addr 172.21.15.117:0/1443450206 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e3fff640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e40c61d0 0x7fa3e414f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e917c640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa3e415ef80 0x7fa3e40ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 -- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa3e415ef80 msgr2=0x7fa3e40ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa3e415ef80 0x7fa3e40ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 -- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e40c61d0 msgr2=0x7fa3e414f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e40c61d0 0x7fa3e414f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.131+0000 7fa3e897b640 1 -- 172.21.15.117:0/1443450206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e414fbe0 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.132+0000 7fa3e897b640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 0x7fa3e41f55a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800c730 tx=0x7fa3d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:56.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.133+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8016020 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.133+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e40d0120 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.133+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e40d06e0 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.134+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3d800a950 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.134+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e40cdf40 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.135+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d800ada0 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.137+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa3d8019020 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.138+0000 7fa3e17fa640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa3c8078b90 0x7fa3c807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:56.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.138+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fa3d809b560 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.138+0000 7fa3e3fff640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa3c8078b90 0x7fa3c807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:56.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.139+0000 7fa3e3fff640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa3c8078b90 0x7fa3c807b0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0034000 tx=0x7fa3d00046c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:56.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.144+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa3d8020020 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.597+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3e415ad30 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.598+0000 7fa3e17fa640 1 -- 172.21.15.117:0/1443450206 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7fa3e40c74f0 con 0x7fa3e40c5ad0 2024-09-17T05:31:56.600 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:56.600 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:31:56.600 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:31:56.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa3c8078b90 msgr2=0x7fa3c807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:56.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa3c8078b90 0x7fa3c807b0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0034000 tx=0x7fa3d00046c0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 msgr2=0x7fa3e41f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 0x7fa3e41f55a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800c730 tx=0x7fa3d800cbf0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 shutdown_connections 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa3c8078b90 0x7fa3c807b0a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa3e415ef80 0x7fa3e40ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3e40c61d0 0x7fa3e414f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 --2- 172.21.15.117:0/1443450206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3e40c5ad0 0x7fa3e41f55a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.603+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 >> 172.21.15.117:0/1443450206 conn(0x7fa3e406e020 msgr2=0x7fa3e40b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:56.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.604+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 shutdown_connections 2024-09-17T05:31:56.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:56.604+0000 7fa3eabdf640 1 -- 172.21.15.117:0/1443450206 wait complete. 2024-09-17T05:31:56.626 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:31:58.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 -- 172.21.15.117:0/1521800635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 msgr2=0x7f9cd80c3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/1521800635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80c3820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80c4750 tx=0x7f9cd002a940 comp rx=0 tx=0).stop 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 -- 172.21.15.117:0/1521800635 shutdown_connections 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/1521800635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80c3820 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/1521800635 >> v2:172.21.15.119:3300/0 conn(0x7f9cd80c8810 0x7f9cd80c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 -- 172.21.15.117:0/1521800635 >> 172.21.15.117:0/1521800635 conn(0x7f9cd806d190 msgr2=0x7f9cd806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:58.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 -- 172.21.15.117:0/1521800635 shutdown_connections 2024-09-17T05:31:58.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.750+0000 7f9cdda7e640 1 -- 172.21.15.117:0/1521800635 wait complete. 2024-09-17T05:31:58.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.753+0000 7f9cdda7e640 1 Processor -- start 2024-09-17T05:31:58.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.753+0000 7f9cdda7e640 1 -- start start 2024-09-17T05:31:58.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cdda7e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:58.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cdda7e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9cd80c63e0 0x7f9cd80ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:58.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cdda7e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cd80c8810 0x7f9cd8142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cdda7e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd8073f60 con 0x7f9cd8073b50 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cdda7e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd8074e30 con 0x7f9cd80c8810 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cdda7e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd80c9710 con 0x7f9cd80c63e0 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:33206/0 (socket says 172.21.15.117:33206) 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 -- 172.21.15.117:0/3419114438 learned_addr learned my addr 172.21.15.117:0/3419114438 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd7fff640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cd80c8810 0x7f9cd8142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9ccffff640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9cd80c63e0 0x7f9cd80ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 -- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9cd80c63e0 msgr2=0x7f9cd80ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9cd80c63e0 0x7f9cd80ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 -- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cd80c8810 msgr2=0x7f9cd8142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cd80c8810 0x7f9cd8142c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:58.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.755+0000 7f9cd77fe640 1 -- 172.21.15.117:0/3419114438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd80cbd20 con 0x7f9cd8073b50 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.756+0000 7f9cd77fe640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80cca90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800c430 tx=0x7f9cc800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.756+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8011790 con 0x7f9cd8073b50 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.756+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cc8017070 con 0x7f9cd8073b50 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.756+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8011ce0 con 0x7f9cd8073b50 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.756+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd80cc2d0 con 0x7f9cd8073b50 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.756+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd8143c60 con 0x7f9cd8073b50 2024-09-17T05:31:58.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.757+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd80c9b40 con 0x7f9cd8073b50 2024-09-17T05:31:58.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.758+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9cc801c460 con 0x7f9cd8073b50 2024-09-17T05:31:58.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.759+0000 7f9cd4ff9640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9cb807ada0 0x7f9cb807d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:31:58.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.759+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f9cc809ae80 con 0x7f9cd8073b50 2024-09-17T05:31:58.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.759+0000 7f9ccffff640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9cb807ada0 0x7f9cb807d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:31:58.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.760+0000 7f9ccffff640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9cb807ada0 0x7f9cb807d2b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9cd815a450 tx=0x7f9cc0005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:31:58.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:58.764+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9cc8062850 con 0x7f9cd8073b50 2024-09-17T05:31:59.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.026+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cd8143ea0 con 0x7f9cd8073b50 2024-09-17T05:31:59.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.027+0000 7f9cd4ff9640 1 -- 172.21.15.117:0/3419114438 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7f9cc8067280 con 0x7f9cd8073b50 2024-09-17T05:31:59.028 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:31:59.028 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:31:59.029 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:31:59.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9cb807ada0 msgr2=0x7f9cb807d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9cb807ada0 0x7f9cb807d2b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9cd815a450 tx=0x7f9cc0005f90 comp rx=0 tx=0).stop 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 msgr2=0x7f9cd80cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80cca90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800c430 tx=0x7f9cc800c8f0 comp rx=0 tx=0).stop 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 shutdown_connections 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9cb807ada0 0x7f9cb807d2b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cd80c8810 0x7f9cd8142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9cd80c63e0 0x7f9cd80ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 --2- 172.21.15.117:0/3419114438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cd8073b50 0x7f9cd80cca90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 >> 172.21.15.117:0/3419114438 conn(0x7f9cd806d190 msgr2=0x7f9cd814e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 shutdown_connections 2024-09-17T05:31:59.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:31:59.029+0000 7f9cdda7e640 1 -- 172.21.15.117:0/3419114438 wait complete. 2024-09-17T05:31:59.042 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:32:01.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:01.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.171+0000 7f7143d2b640 1 -- 172.21.15.117:0/2020908700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7128004d40 msgr2=0x7f7128005170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:01.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.171+0000 7f7143d2b640 1 --2- 172.21.15.117:0/2020908700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7128004d40 0x7f7128005170 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7134001f30 tx=0x7f7128005830 comp rx=0 tx=0).stop 2024-09-17T05:32:01.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.171+0000 7f71422c8640 1 -- 172.21.15.117:0/2020908700 reap_dead start 2024-09-17T05:32:01.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.171+0000 7f7143d2b640 1 -- 172.21.15.117:0/2020908700 shutdown_connections 2024-09-17T05:32:01.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.171+0000 7f7143d2b640 1 -- 172.21.15.117:0/2020908700 >> 172.21.15.117:0/2020908700 conn(0x7f713c06e590 msgr2=0x7f713c0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:01.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.173+0000 7f7143d2b640 1 -- 172.21.15.117:0/2020908700 shutdown_connections 2024-09-17T05:32:01.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.173+0000 7f7143d2b640 1 -- 172.21.15.117:0/2020908700 wait complete. 2024-09-17T05:32:01.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.175+0000 7f7143d2b640 1 Processor -- start 2024-09-17T05:32:01.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.175+0000 7f7143d2b640 1 -- start start 2024-09-17T05:32:01.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7143d2b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7128004d40 0x7f713c0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:01.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7143d2b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 0x7f713c146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7143d2b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c14d590 0x7f713c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7143d2b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f713c0ce0d0 con 0x7f713c14d590 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7143d2b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f713c0c58a0 con 0x7f713c06d5f0 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7143d2b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f713c0c7ff0 con 0x7f7128004d40 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71412c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 0x7f713c146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71412c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 0x7f713c146330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:54636/0 (socket says 172.21.15.117:54636) 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71412c6640 1 -- 172.21.15.117:0/1362879938 learned_addr learned my addr 172.21.15.117:0/1362879938 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71422c8640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c14d590 0x7f713c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f7141ac7640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7128004d40 0x7f713c0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71412c6640 1 -- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7128004d40 msgr2=0x7f713c0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71412c6640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7128004d40 0x7f713c0cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.177+0000 7f71412c6640 1 -- 172.21.15.117:0/1362879938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c14d590 msgr2=0x7f713c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71412c6640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c14d590 0x7f713c14d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71412c6640 1 -- 172.21.15.117:0/1362879938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c157250 con 0x7f713c06d5f0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71422c8640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c14d590 0x7f713c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71412c6640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 0x7f713c146330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f712c00e970 tx=0x7f712c00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c00ccc0 con 0x7f713c06d5f0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f7143d2b640 1 -- 172.21.15.117:0/1362879938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713c157530 con 0x7f713c06d5f0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f712c00ce60 con 0x7f713c06d5f0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c005280 con 0x7f713c06d5f0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.178+0000 7f7143d2b640 1 -- 172.21.15.117:0/1362879938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f713c0cb9d0 con 0x7f713c06d5f0 2024-09-17T05:32:01.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.180+0000 7f7143d2b640 1 -- 172.21.15.117:0/1362879938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713c0c8d00 con 0x7f713c06d5f0 2024-09-17T05:32:01.182 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.181+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f712c005420 con 0x7f713c06d5f0 2024-09-17T05:32:01.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.182+0000 7f71327fc640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7120078bc0 0x7f712007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:01.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.182+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f712c09bad0 con 0x7f713c06d5f0 2024-09-17T05:32:01.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.183+0000 7f7141ac7640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7120078bc0 0x7f712007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:01.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.184+0000 7f7141ac7640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7120078bc0 0x7f712007b0d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7138006be0 tx=0x7f7138000fe0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:01.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.191+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f712c063b90 con 0x7f713c06d5f0 2024-09-17T05:32:01.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.646+0000 7f7143d2b640 1 -- 172.21.15.117:0/1362879938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f713c0c4750 con 0x7f713c06d5f0 2024-09-17T05:32:01.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.651+0000 7f71327fc640 1 -- 172.21.15.117:0/1362879938 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7f712c063490 con 0x7f713c06d5f0 2024-09-17T05:32:01.656 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:01.656 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:32:01.657 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 -- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7120078bc0 msgr2=0x7f712007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7120078bc0 0x7f712007b0d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7138006be0 tx=0x7f7138000fe0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 -- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 msgr2=0x7f713c146330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 0x7f713c146330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f712c00e970 tx=0x7f712c00ee30 comp rx=0 tx=0).stop 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 -- 172.21.15.117:0/1362879938 shutdown_connections 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f7120078bc0 0x7f712007b0d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713c14d590 0x7f713c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c06d5f0 0x7f713c146330 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 --2- 172.21.15.117:0/1362879938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7128004d40 0x7f713c0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.655+0000 7f711ffff640 1 -- 172.21.15.117:0/1362879938 >> 172.21.15.117:0/1362879938 conn(0x7f713c06e590 msgr2=0x7f713c158af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:01.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.656+0000 7f711ffff640 1 -- 172.21.15.117:0/1362879938 shutdown_connections 2024-09-17T05:32:01.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:01.656+0000 7f711ffff640 1 -- 172.21.15.117:0/1362879938 wait complete. 2024-09-17T05:32:01.680 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:31:45:857416+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4434": { "addr": "172.21.15.119:6835/2224210538", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2224210538, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2224210538, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4434, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:31:45.857410+0000", "qdb_cluster": [ 4434, 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 4434, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:32:03.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:03.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.873+0000 7f8240c9b640 1 -- 172.21.15.117:0/2053663180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 msgr2=0x7f823c15ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:03.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.873+0000 7f8240c9b640 1 --2- 172.21.15.117:0/2053663180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c15ae60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8234009be0 tx=0x7f823402a8d0 comp rx=0 tx=0).stop 2024-09-17T05:32:03.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.874+0000 7f8240c9b640 1 -- 172.21.15.117:0/2053663180 shutdown_connections 2024-09-17T05:32:03.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.874+0000 7f8240c9b640 1 --2- 172.21.15.117:0/2053663180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c15ae60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:03.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.874+0000 7f8240c9b640 1 -- 172.21.15.117:0/2053663180 >> 172.21.15.117:0/2053663180 conn(0x7f823c06d190 msgr2=0x7f823c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:03.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.876+0000 7f8240c9b640 1 -- 172.21.15.117:0/2053663180 shutdown_connections 2024-09-17T05:32:03.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.876+0000 7f8240c9b640 1 -- 172.21.15.117:0/2053663180 wait complete. 2024-09-17T05:32:03.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.880+0000 7f8240c9b640 1 Processor -- start 2024-09-17T05:32:03.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.881+0000 7f8240c9b640 1 -- start start 2024-09-17T05:32:03.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.884+0000 7f8240c9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f823c072e10 0x7f823c159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:03.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.884+0000 7f8240c9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:03.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.884+0000 7f8240c9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823c157ea0 0x7f823c15a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:03.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.884+0000 7f8240c9b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f823c0c67f0 con 0x7f823c0c6ca0 2024-09-17T05:32:03.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.884+0000 7f8240c9b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f823c0c3820 con 0x7f823c157ea0 2024-09-17T05:32:03.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.884+0000 7f8240c9b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f823c0c9b20 con 0x7f823c072e10 2024-09-17T05:32:03.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.885+0000 7f8231d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:03.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.885+0000 7f8231d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:33244/0 (socket says 172.21.15.117:33244) 2024-09-17T05:32:03.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.885+0000 7f8231d9b640 1 -- 172.21.15.117:0/3837740094 learned_addr learned my addr 172.21.15.117:0/3837740094 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:03.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f823a59c640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f823c072e10 0x7f823c159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:03.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f823ad9d640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823c157ea0 0x7f823c15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f8231d9b640 1 -- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f823c072e10 msgr2=0x7f823c159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f8231d9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f823c072e10 0x7f823c159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f8231d9b640 1 -- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823c157ea0 msgr2=0x7f823c15a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f8231d9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823c157ea0 0x7f823c15a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.886+0000 7f8231d9b640 1 -- 172.21.15.117:0/3837740094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c1410b0 con 0x7f823c0c6ca0 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.887+0000 7f8231d9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c159dc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f822c00b3e0 tx=0x7f822c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.887+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c015020 con 0x7f823c0c6ca0 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.887+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f822c011920 con 0x7f823c0c6ca0 2024-09-17T05:32:03.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.888+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c011d30 con 0x7f823c0c6ca0 2024-09-17T05:32:03.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.890+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823c151970 con 0x7f823c0c6ca0 2024-09-17T05:32:03.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.891+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f823c13e890 con 0x7f823c0c6ca0 2024-09-17T05:32:03.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.894+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f822c018020 con 0x7f823c0c6ca0 2024-09-17T05:32:03.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.895+0000 7f82337fe640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8220078c60 0x7f822007b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:03.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.895+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f822c09ae20 con 0x7f823c0c6ca0 2024-09-17T05:32:03.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.895+0000 7f823a59c640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8220078c60 0x7f822007b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:03.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.896+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823c151610 con 0x7f823c0c6ca0 2024-09-17T05:32:03.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.900+0000 7f823a59c640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8220078c60 0x7f822007b170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f82240046b0 tx=0x7f8224007e90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:03.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:03.906+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f822c0627d0 con 0x7f823c0c6ca0 2024-09-17T05:32:04.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.342+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f823c075180 con 0x7f823c0c6ca0 2024-09-17T05:32:04.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.342+0000 7f82337fe640 1 -- 172.21.15.117:0/3837740094 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6096 (secure 0 0 0) 0x7f822c067200 con 0x7f823c0c6ca0 2024-09-17T05:32:04.344 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:04.344 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":21,"btime":"2024-09-17T05:31:45:857416+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:31:45.857410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:32:04.344 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 21 2024-09-17T05:32:04.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.346+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8220078c60 msgr2=0x7f822007b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:04.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.346+0000 7f8240c9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8220078c60 0x7f822007b170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f82240046b0 tx=0x7f8224007e90 comp rx=0 tx=0).stop 2024-09-17T05:32:04.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.346+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 msgr2=0x7f823c159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:04.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.346+0000 7f8240c9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c159dc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f822c00b3e0 tx=0x7f822c00b8a0 comp rx=0 tx=0).stop 2024-09-17T05:32:04.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.347+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 shutdown_connections 2024-09-17T05:32:04.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.347+0000 7f8240c9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8220078c60 0x7f822007b170 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f82240046b0 tx=0x7f8224007e90 comp rx=0 tx=0).stop 2024-09-17T05:32:04.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.347+0000 7f8240c9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823c157ea0 0x7f823c15a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:04.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.347+0000 7f8240c9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f823c0c6ca0 0x7f823c159dc0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f822c00b3e0 tx=0x7f822c00b8a0 comp rx=0 tx=0).stop 2024-09-17T05:32:04.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.347+0000 7f8240c9b640 1 --2- 172.21.15.117:0/3837740094 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f823c072e10 0x7f823c159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:04.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.347+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 >> 172.21.15.117:0/3837740094 conn(0x7f823c06d190 msgr2=0x7f823c156430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:04.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.349+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 shutdown_connections 2024-09-17T05:32:04.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:04.349+0000 7f8240c9b640 1 -- 172.21.15.117:0/3837740094 wait complete. 2024-09-17T05:32:04.370 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:32:06.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:06.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.572+0000 7f228e0ec640 1 -- 172.21.15.117:0/4207088773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2288153260 msgr2=0x7f228815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:06.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.572+0000 7f228e0ec640 1 --2- 172.21.15.117:0/4207088773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2288153260 0x7f228815d220 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2288074aa0 tx=0x7f227c02a580 comp rx=0 tx=0).stop 2024-09-17T05:32:06.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.573+0000 7f228e0ec640 1 -- 172.21.15.117:0/4207088773 shutdown_connections 2024-09-17T05:32:06.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.573+0000 7f228e0ec640 1 --2- 172.21.15.117:0/4207088773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2288153260 0x7f228815d220 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2288074aa0 tx=0x7f227c02a580 comp rx=0 tx=0).stop 2024-09-17T05:32:06.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.573+0000 7f228e0ec640 1 --2- 172.21.15.117:0/4207088773 >> v2:172.21.15.119:3300/0 conn(0x7f22880cad30 0x7f22880c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:06.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.573+0000 7f228e0ec640 1 -- 172.21.15.117:0/4207088773 >> 172.21.15.117:0/4207088773 conn(0x7f2288076430 msgr2=0x7f2288072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:06.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.575+0000 7f228e0ec640 1 -- 172.21.15.117:0/4207088773 shutdown_connections 2024-09-17T05:32:06.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.577+0000 7f228e0ec640 1 -- 172.21.15.117:0/4207088773 wait complete. 2024-09-17T05:32:06.581 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.580+0000 7f228e0ec640 1 Processor -- start 2024-09-17T05:32:06.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.580+0000 7f228e0ec640 1 -- start start 2024-09-17T05:32:06.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.582+0000 7f228e0ec640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 0x7f228814ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f22877fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 0x7f228814ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f22877fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 0x7f228814ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58792/0 (socket says 172.21.15.117:58792) 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f22877fe640 1 -- 172.21.15.117:0/3251533908 learned_addr learned my addr 172.21.15.117:0/3251533908 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f228e0ec640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2288159b30 0x7f228814acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f228e0ec640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2288072c70 0x7f228814e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f228e0ec640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2288159f40 con 0x7f22880cad30 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f228e0ec640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f22880cbc30 con 0x7f2288159b30 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f228e0ec640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f22881eda30 con 0x7f2288072c70 2024-09-17T05:32:06.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.583+0000 7f2286ffd640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2288159b30 0x7f228814acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:06.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.584+0000 7f22877fe640 1 -- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2288072c70 msgr2=0x7f228814e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:06.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.584+0000 7f22877fe640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2288072c70 0x7f228814e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:06.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.584+0000 7f22877fe640 1 -- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2288159b30 msgr2=0x7f228814acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:06.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.584+0000 7f22877fe640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2288159b30 0x7f228814acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:06.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.584+0000 7f22877fe640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228814b200 con 0x7f22880cad30 2024-09-17T05:32:06.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.584+0000 7f22877fe640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 0x7f228814ccd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f227800d740 tx=0x7f227800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:06.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.585+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278017070 con 0x7f22880cad30 2024-09-17T05:32:06.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.585+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2278009d20 con 0x7f22880cad30 2024-09-17T05:32:06.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.585+0000 7f228e0ec640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228806fa50 con 0x7f22880cad30 2024-09-17T05:32:06.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.587+0000 7f228e0ec640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228815ac60 con 0x7f22880cad30 2024-09-17T05:32:06.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.591+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780106c0 con 0x7f22880cad30 2024-09-17T05:32:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.593+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f227801a070 con 0x7f22880cad30 2024-09-17T05:32:06.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.594+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2268005290 con 0x7f22880cad30 2024-09-17T05:32:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.594+0000 7f226ffff640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2274078c60 0x7f227407b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.594+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f227809b2c0 con 0x7f22880cad30 2024-09-17T05:32:06.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.596+0000 7f2286ffd640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2274078c60 0x7f227407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:06.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.597+0000 7f2286ffd640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2274078c60 0x7f227407b170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f227c033000 tx=0x7f227c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:06.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:06.604+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2278062c70 con 0x7f22880cad30 2024-09-17T05:32:07.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.048+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2268005580 con 0x7f22880cad30 2024-09-17T05:32:07.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.050+0000 7f226ffff640 1 -- 172.21.15.117:0/3251533908 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6098 (secure 0 0 0) 0x7f22780676a0 con 0x7f22880cad30 2024-09-17T05:32:07.054 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:07.054 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":22,"btime":"2024-09-17T05:32:04:912806+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:04.667893+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":40,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:32:07.054 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 22 2024-09-17T05:32:07.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2274078c60 msgr2=0x7f227407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:07.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2274078c60 0x7f227407b170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f227c033000 tx=0x7f227c004730 comp rx=0 tx=0).stop 2024-09-17T05:32:07.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 msgr2=0x7f228814ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:07.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 0x7f228814ccd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f227800d740 tx=0x7f227800dc00 comp rx=0 tx=0).stop 2024-09-17T05:32:07.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 shutdown_connections 2024-09-17T05:32:07.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2274078c60 0x7f227407b170 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:07.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2288072c70 0x7f228814e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:07.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.056+0000 7f226dffb640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2288159b30 0x7f228814acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:07.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.057+0000 7f226dffb640 1 --2- 172.21.15.117:0/3251533908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880cad30 0x7f228814ccd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:07.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.057+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 >> 172.21.15.117:0/3251533908 conn(0x7f2288076430 msgr2=0x7f22881522f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:07.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.057+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 shutdown_connections 2024-09-17T05:32:07.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:07.057+0000 7f226dffb640 1 -- 172.21.15.117:0/3251533908 wait complete. 2024-09-17T05:32:07.081 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:32:09.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:09.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 -- 172.21.15.117:0/1571431382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 msgr2=0x7f90180ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:09.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 --2- 172.21.15.117:0/1571431382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 0x7f90180ca770 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f901808e1f0 tx=0x7f900c02a680 comp rx=0 tx=0).stop 2024-09-17T05:32:09.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 -- 172.21.15.117:0/1571431382 shutdown_connections 2024-09-17T05:32:09.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 --2- 172.21.15.117:0/1571431382 >> v2:172.21.15.119:3300/0 conn(0x7f90180b56c0 0x7f9018165c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 --2- 172.21.15.117:0/1571431382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 0x7f90180ca770 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 -- 172.21.15.117:0/1571431382 >> 172.21.15.117:0/1571431382 conn(0x7f9018074d80 msgr2=0x7f90180d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:09.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 -- 172.21.15.117:0/1571431382 shutdown_connections 2024-09-17T05:32:09.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.216+0000 7f901eb62640 1 -- 172.21.15.117:0/1571431382 wait complete. 2024-09-17T05:32:09.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.218+0000 7f901eb62640 1 Processor -- start 2024-09-17T05:32:09.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.218+0000 7f901eb62640 1 -- start start 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901eb62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 0x7f90181f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901eb62640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 0x7f90181585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901eb62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 0x7f901814fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901eb62640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f90180cacb0 con 0x7f901815bd70 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901eb62640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f90180cd0d0 con 0x7f90180b52b0 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901eb62640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f90181558b0 con 0x7f90180b56c0 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901c8fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 0x7f90181f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 0x7f90181585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901c8fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 0x7f90181f1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:38916/0 (socket says 172.21.15.117:38916) 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 0x7f90181585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:55890/0 (socket says 172.21.15.117:55890) 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901c8fe640 1 -- 172.21.15.117:0/1623709731 learned_addr learned my addr 172.21.15.117:0/1623709731 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:09.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901d0ff640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 0x7f901814fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 -- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 msgr2=0x7f90181f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 0x7f90181f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 -- 172.21.15.117:0/1623709731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 msgr2=0x7f901814fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 0x7f901814fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f9017fff640 1 -- 172.21.15.117:0/1623709731 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90181f2370 con 0x7f90180b56c0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.221+0000 7f901c8fe640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 0x7f90181f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.222+0000 7f9017fff640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 0x7f90181585b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f900400c3f0 tx=0x7f900400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.222+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004017070 con 0x7f90180b56c0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.222+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90040053c0 con 0x7f90180b56c0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.222+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9018159500 con 0x7f90180b56c0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.222+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9018156500 con 0x7f90180b56c0 2024-09-17T05:32:09.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.222+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004003850 con 0x7f90180b56c0 2024-09-17T05:32:09.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.223+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8005290 con 0x7f90180b56c0 2024-09-17T05:32:09.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.224+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f900401a070 con 0x7f90180b56c0 2024-09-17T05:32:09.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.224+0000 7f90157fa640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8ffc078bc0 0x7f8ffc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:09.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.224+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f900409a310 con 0x7f90180b56c0 2024-09-17T05:32:09.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.224+0000 7f901c8fe640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8ffc078bc0 0x7f8ffc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:09.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.225+0000 7f901c8fe640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8ffc078bc0 0x7f8ffc07b0d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f900c009230 tx=0x7f900c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:09.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.229+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9004061ce0 con 0x7f90180b56c0 2024-09-17T05:32:09.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.671+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ff8005850 con 0x7f90180b56c0 2024-09-17T05:32:09.677 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:09.677 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":22,"btime":"2024-09-17T05:32:04:912806+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:04.667893+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:resolve","state_seq":40,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:32:09.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.674+0000 7f90157fa640 1 -- 172.21.15.117:0/1623709731 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6098 (secure 0 0 0) 0x7f9004066710 con 0x7f90180b56c0 2024-09-17T05:32:09.677 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 22 2024-09-17T05:32:09.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.678+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8ffc078bc0 msgr2=0x7f8ffc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:09.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.678+0000 7f901eb62640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8ffc078bc0 0x7f8ffc07b0d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f900c009230 tx=0x7f900c004730 comp rx=0 tx=0).stop 2024-09-17T05:32:09.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.678+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 msgr2=0x7f90181585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:09.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.678+0000 7f901eb62640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 0x7f90181585b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f900400c3f0 tx=0x7f900400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.679+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 shutdown_connections 2024-09-17T05:32:09.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.679+0000 7f901eb62640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8ffc078bc0 0x7f8ffc07b0d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.679+0000 7f901eb62640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f901815bd70 0x7f901814fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.679+0000 7f901eb62640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f90180b56c0 0x7f90181585b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.679+0000 7f901eb62640 1 --2- 172.21.15.117:0/1623709731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90180b52b0 0x7f90181f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:09.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.679+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 >> 172.21.15.117:0/1623709731 conn(0x7f9018074d80 msgr2=0x7f90180c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:09.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.682+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 shutdown_connections 2024-09-17T05:32:09.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:09.682+0000 7f901eb62640 1 -- 172.21.15.117:0/1623709731 wait complete. 2024-09-17T05:32:09.705 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:32:11.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:11.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.842+0000 7f68ef069640 1 -- 172.21.15.117:0/172864512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e8152210 msgr2=0x7f68e80c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.842+0000 7f68ef069640 1 --2- 172.21.15.117:0/172864512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e8152210 0x7f68e80c4f50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f68e80ce040 tx=0x7f68d402a410 comp rx=0 tx=0).stop 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.843+0000 7f68ef069640 1 -- 172.21.15.117:0/172864512 shutdown_connections 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.843+0000 7f68ef069640 1 --2- 172.21.15.117:0/172864512 >> v2:172.21.15.119:3300/0 conn(0x7f68e80c5490 0x7f68e806cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.843+0000 7f68ef069640 1 --2- 172.21.15.117:0/172864512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e8152210 0x7f68e80c4f50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.843+0000 7f68ef069640 1 -- 172.21.15.117:0/172864512 >> 172.21.15.117:0/172864512 conn(0x7f68e806e590 msgr2=0x7f68e80cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.843+0000 7f68ef069640 1 -- 172.21.15.117:0/172864512 shutdown_connections 2024-09-17T05:32:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.843+0000 7f68ef069640 1 -- 172.21.15.117:0/172864512 wait complete. 2024-09-17T05:32:11.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.845+0000 7f68ef069640 1 Processor -- start 2024-09-17T05:32:11.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.845+0000 7f68ef069640 1 -- start start 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ef069640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e80c5490 0x7f68e8151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ef069640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 0x7f68e80c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ef069640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e8152210 0x7f68e814b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ef069640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f68e80ce0d0 con 0x7f68e80c67a0 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ef069640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f68e80c58a0 con 0x7f68e8152210 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ef069640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f68e80c7ff0 con 0x7f68e80c5490 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68e7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 0x7f68e80c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68e7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 0x7f68e80c7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58824/0 (socket says 172.21.15.117:58824) 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68e7fff640 1 -- 172.21.15.117:0/3286169393 learned_addr learned my addr 172.21.15.117:0/3286169393 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ed606640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e8152210 0x7f68e814b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.846+0000 7f68ece05640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e80c5490 0x7f68e8151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e7fff640 1 -- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e80c5490 msgr2=0x7f68e8151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e7fff640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e80c5490 0x7f68e8151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e7fff640 1 -- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e8152210 msgr2=0x7f68e814b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e7fff640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e8152210 0x7f68e814b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e7fff640 1 -- 172.21.15.117:0/3286169393 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e80cd3c0 con 0x7f68e80c67a0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e7fff640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 0x7f68e80c7170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009920 tx=0x7f68d4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4031d60 con 0x7f68e80c67a0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68d4033e10 con 0x7f68e80c67a0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4005260 con 0x7f68e80c67a0 2024-09-17T05:32:11.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e80cc780 con 0x7f68e80c67a0 2024-09-17T05:32:11.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.847+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e8150c40 con 0x7f68e80c67a0 2024-09-17T05:32:11.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.849+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e80c8d00 con 0x7f68e80c67a0 2024-09-17T05:32:11.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.849+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f68d4005400 con 0x7f68e80c67a0 2024-09-17T05:32:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.850+0000 7f68e57fa640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68cc078b90 0x7f68cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.850+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f68d40bc120 con 0x7f68e80c67a0 2024-09-17T05:32:11.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.853+0000 7f68ece05640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68cc078b90 0x7f68cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:11.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.854+0000 7f68ece05640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68cc078b90 0x7f68cc07b0a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f68dc000f60 tx=0x7f68dc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:11.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:11.856+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f68d403f030 con 0x7f68e80c67a0 2024-09-17T05:32:12.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.110+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68e80cc4d0 con 0x7f68e80c67a0 2024-09-17T05:32:12.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.111+0000 7f68e57fa640 1 -- 172.21.15.117:0/3286169393 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6100 (secure 0 0 0) 0x7f68d4083c10 con 0x7f68e80c67a0 2024-09-17T05:32:12.112 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:12.112 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":23,"btime":"2024-09-17T05:32:09:954455+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:09.916005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":42,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:32:12.113 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 23 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68cc078b90 msgr2=0x7f68cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68cc078b90 0x7f68cc07b0a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f68dc000f60 tx=0x7f68dc009040 comp rx=0 tx=0).stop 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 msgr2=0x7f68e80c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 0x7f68e80c7170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009920 tx=0x7f68d4033c20 comp rx=0 tx=0).stop 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 shutdown_connections 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68cc078b90 0x7f68cc07b0a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:12.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e8152210 0x7f68e814b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:12.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e80c67a0 0x7f68e80c7170 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:12.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 --2- 172.21.15.117:0/3286169393 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e80c5490 0x7f68e8151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:12.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 >> 172.21.15.117:0/3286169393 conn(0x7f68e806e590 msgr2=0x7f68e80ca970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:12.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 shutdown_connections 2024-09-17T05:32:12.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:12.113+0000 7f68ef069640 1 -- 172.21.15.117:0/3286169393 wait complete. 2024-09-17T05:32:12.126 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:32:14.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:14.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.304+0000 7f30454cc640 1 -- 172.21.15.117:0/2043635053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3040074e60 msgr2=0x7f30400c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:14.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.304+0000 7f30454cc640 1 --2- 172.21.15.117:0/2043635053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3040074e60 0x7f30400c96d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f30400724d0 tx=0x7f303802bad0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.305+0000 7f30454cc640 1 -- 172.21.15.117:0/2043635053 shutdown_connections 2024-09-17T05:32:14.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.305+0000 7f30454cc640 1 --2- 172.21.15.117:0/2043635053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3040074e60 0x7f30400c96d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.305+0000 7f30454cc640 1 --2- 172.21.15.117:0/2043635053 >> v2:172.21.15.119:3300/0 conn(0x7f304008c4c0 0x7f30400c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.306+0000 7f30454cc640 1 -- 172.21.15.117:0/2043635053 >> 172.21.15.117:0/2043635053 conn(0x7f304006d190 msgr2=0x7f304006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:14.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.307+0000 7f30454cc640 1 -- 172.21.15.117:0/2043635053 shutdown_connections 2024-09-17T05:32:14.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.307+0000 7f30454cc640 1 -- 172.21.15.117:0/2043635053 wait complete. 2024-09-17T05:32:14.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.311+0000 7f30454cc640 1 Processor -- start 2024-09-17T05:32:14.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.311+0000 7f30454cc640 1 -- start start 2024-09-17T05:32:14.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.313+0000 7f30454cc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3040074e60 0x7f30401602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.313+0000 7f30454cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 0x7f3040169ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.313+0000 7f30454cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30400c92c0 0x7f3040209a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.313+0000 7f30454cc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f30400c8b80 con 0x7f304008c4c0 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.313+0000 7f30454cc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f304016b5d0 con 0x7f30400c92c0 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.313+0000 7f30454cc640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f30400c8d00 con 0x7f3040074e60 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 0x7f3040169ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 0x7f3040169ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:46902/0 (socket says 172.21.15.117:46902) 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303e7fc640 1 -- 172.21.15.117:0/2490905293 learned_addr learned my addr 172.21.15.117:0/2490905293 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303f7fe640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30400c92c0 0x7f3040209a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:14.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303effd640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3040074e60 0x7f30401602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:14.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303e7fc640 1 -- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3040074e60 msgr2=0x7f30401602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:14.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.314+0000 7f303e7fc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3040074e60 0x7f30401602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.315+0000 7f303e7fc640 1 -- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30400c92c0 msgr2=0x7f3040209a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:14.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.315+0000 7f303e7fc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30400c92c0 0x7f3040209a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.315+0000 7f303e7fc640 1 -- 172.21.15.117:0/2490905293 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3040160810 con 0x7f304008c4c0 2024-09-17T05:32:14.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.315+0000 7f303e7fc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 0x7f3040169ff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f303400c340 tx=0x7f303400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:14.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.316+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340052d0 con 0x7f304008c4c0 2024-09-17T05:32:14.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.316+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3034007020 con 0x7f304008c4c0 2024-09-17T05:32:14.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.316+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034007470 con 0x7f304008c4c0 2024-09-17T05:32:14.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.317+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304015f530 con 0x7f304008c4c0 2024-09-17T05:32:14.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.317+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304015f7f0 con 0x7f304008c4c0 2024-09-17T05:32:14.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.319+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3020005290 con 0x7f304008c4c0 2024-09-17T05:32:14.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.322+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f30340067d0 con 0x7f304008c4c0 2024-09-17T05:32:14.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.323+0000 7f30277fe640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f302c078c60 0x7f302c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:14.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.323+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f303409a960 con 0x7f304008c4c0 2024-09-17T05:32:14.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.324+0000 7f303effd640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f302c078c60 0x7f302c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:14.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.327+0000 7f303effd640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f302c078c60 0x7f302c07b170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f303000bb50 tx=0x7f303000b2e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:14.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.329+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f30340623a0 con 0x7f304008c4c0 2024-09-17T05:32:14.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.787+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3020005580 con 0x7f304008c4c0 2024-09-17T05:32:14.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.791+0000 7f30277fe640 1 -- 172.21.15.117:0/2490905293 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6100 (secure 0 0 0) 0x7f3034066dd0 con 0x7f304008c4c0 2024-09-17T05:32:14.793 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:14.793 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":23,"btime":"2024-09-17T05:32:09:954455+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:09.916005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:reconnect","state_seq":42,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440]},"id":1}]} 2024-09-17T05:32:14.794 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 23 2024-09-17T05:32:14.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f302c078c60 msgr2=0x7f302c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:14.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f302c078c60 0x7f302c07b170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f303000bb50 tx=0x7f303000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 msgr2=0x7f3040169ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:14.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 0x7f3040169ff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f303400c340 tx=0x7f303400c800 comp rx=0 tx=0).stop 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 shutdown_connections 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f302c078c60 0x7f302c07b170 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f303000bb50 tx=0x7f303000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30400c92c0 0x7f3040209a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f304008c4c0 0x7f3040169ff0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 --2- 172.21.15.117:0/2490905293 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3040074e60 0x7f30401602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.796+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 >> 172.21.15.117:0/2490905293 conn(0x7f304006d190 msgr2=0x7f3040166330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.797+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 shutdown_connections 2024-09-17T05:32:14.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:14.797+0000 7f30454cc640 1 -- 172.21.15.117:0/2490905293 wait complete. 2024-09-17T05:32:14.818 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:32:09:954455+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4434": { "addr": "172.21.15.119:6835/2224210538", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2224210538, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2224210538, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4434, "incarnation": 9, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:32:09.916005+0000", "qdb_cluster": [ 4434, 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 4434, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:32:16.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:17.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.032+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/3058912431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 msgr2=0x7fe1ec15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:17.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.032+0000 7fe1ea7fc640 1 -- 172.21.15.117:0/3058912431 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4034020 con 0x7fe1ec0c6f10 2024-09-17T05:32:17.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.032+0000 7fe1f1e5c640 1 --2- 172.21.15.117:0/3058912431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec15b070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4005fd0 tx=0x7fe1e402a9e0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.033+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/3058912431 shutdown_connections 2024-09-17T05:32:17.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.033+0000 7fe1f1e5c640 1 --2- 172.21.15.117:0/3058912431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec15b070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.033+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/3058912431 >> 172.21.15.117:0/3058912431 conn(0x7fe1ec06fbe0 msgr2=0x7fe1ec0706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:17.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.033+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/3058912431 shutdown_connections 2024-09-17T05:32:17.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.033+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/3058912431 wait complete. 2024-09-17T05:32:17.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.036+0000 7fe1f1e5c640 1 Processor -- start 2024-09-17T05:32:17.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.037+0000 7fe1f1e5c640 1 -- start start 2024-09-17T05:32:17.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.038+0000 7fe1f1e5c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe1ec074240 0x7fe1ec0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:17.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1f1e5c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:17.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1f1e5c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1ec15b5b0 0x7fe1ec0cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:17.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1f1e5c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec0c7ec0 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1f1e5c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec0c7620 con 0x7fe1ec0c6f10 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1f1e5c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe1ec0c81a0 con 0x7fe1ec074240 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1eaffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1eaffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec0cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:43922/0 (socket says 172.21.15.117:43922) 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1eaffd640 1 -- 172.21.15.117:0/1557637285 learned_addr learned my addr 172.21.15.117:0/1557637285 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.039+0000 7fe1ebfff640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1ec15b5b0 0x7fe1ec0cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.040+0000 7fe1ebfff640 1 -- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe1ec074240 msgr2=0x7fe1ec0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:17.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.040+0000 7fe1ebfff640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe1ec074240 0x7fe1ec0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.040+0000 7fe1ebfff640 1 -- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 msgr2=0x7fe1ec0cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:17.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.040+0000 7fe1ebfff640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec0cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.040+0000 7fe1ebfff640 1 -- 172.21.15.117:0/1557637285 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ec0cae60 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.040+0000 7fe1ebfff640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1ec15b5b0 0x7fe1ec0cb8d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec06e270 tx=0x7fe1dc00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:17.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.042+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc017070 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.042+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/1557637285 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ec1fe960 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.043+0000 7fe1f1e5c640 1 -- 172.21.15.117:0/1557637285 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ec0c33e0 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.044+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1dc0035f0 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.044+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc017070 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.044+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1ec148320 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.048+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe1dc0037a0 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.048+0000 7fe1d3fff640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1d8078f90 0x7fe1d807b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:17.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.049+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fe1dc0053a0 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.059+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe1ec148320 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.059+0000 7fe1eb7fe640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1d8078f90 0x7fe1d807b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:17.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.062+0000 7fe1eb7fe640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1d8078f90 0x7fe1d807b4a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00015c0 tx=0x7fe1e00079b0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:17.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.488+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1ec0738e0 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.489+0000 7fe1d3fff640 1 -- 172.21.15.117:0/1557637285 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6097 (secure 0 0 0) 0x7fe1dc020020 con 0x7fe1ec15b5b0 2024-09-17T05:32:17.491 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:17.491 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":24,"btime":"2024-09-17T05:32:14:975615+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:14.965940+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:rejoin","state_seq":44,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,4434]},"id":1}]} 2024-09-17T05:32:17.491 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 24 2024-09-17T05:32:17.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1d8078f90 msgr2=0x7fe1d807b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:17.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1d8078f90 0x7fe1d807b4a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00015c0 tx=0x7fe1e00079b0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1ec15b5b0 msgr2=0x7fe1ec0cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:17.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1ec15b5b0 0x7fe1ec0cb8d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec06e270 tx=0x7fe1dc00c820 comp rx=0 tx=0).stop 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 shutdown_connections 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe1d8078f90 0x7fe1d807b4a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1ec15b5b0 0x7fe1ec0cb8d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1ec0c6f10 0x7fe1ec0cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 --2- 172.21.15.117:0/1557637285 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe1ec074240 0x7fe1ec0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 >> 172.21.15.117:0/1557637285 conn(0x7fe1ec06fbe0 msgr2=0x7fe1ec148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 shutdown_connections 2024-09-17T05:32:17.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:17.492+0000 7fe1d1ffb640 1 -- 172.21.15.117:0/1557637285 wait complete. 2024-09-17T05:32:17.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:32:19.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:19.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.782+0000 7fa7eed72640 1 -- 172.21.15.117:0/2072690440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0004800 msgr2=0x7fa7d0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:19.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.782+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2072690440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0004800 0x7fa7d0004c10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80755a0 tx=0x7fa7d0005830 comp rx=0 tx=0).stop 2024-09-17T05:32:19.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.784+0000 7fa7eed72640 1 -- 172.21.15.117:0/2072690440 shutdown_connections 2024-09-17T05:32:19.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.784+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2072690440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0004800 0x7fa7d0004c10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:19.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.784+0000 7fa7eed72640 1 -- 172.21.15.117:0/2072690440 >> 172.21.15.117:0/2072690440 conn(0x7fa7e806d190 msgr2=0x7fa7e806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:19.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.784+0000 7fa7eed72640 1 -- 172.21.15.117:0/2072690440 shutdown_connections 2024-09-17T05:32:19.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.784+0000 7fa7eed72640 1 -- 172.21.15.117:0/2072690440 wait complete. 2024-09-17T05:32:19.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.787+0000 7fa7eed72640 1 Processor -- start 2024-09-17T05:32:19.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.787+0000 7fa7eed72640 1 -- start start 2024-09-17T05:32:19.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7eed72640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 0x7fa7e814b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:19.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7eed72640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7e8159060 0x7fa7e81f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:19.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7eed72640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa7e814d350 0x7fa7e8159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7eed72640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e80c3820 con 0x7fa7d0004800 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7eed72640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e80747f0 con 0x7fa7e8159060 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7eed72640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e80c67f0 con 0x7fa7e814d350 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7ecb0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 0x7fa7e814b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7ecb0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 0x7fa7e814b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:46924/0 (socket says 172.21.15.117:46924) 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7ecb0e640 1 -- 172.21.15.117:0/2562234151 learned_addr learned my addr 172.21.15.117:0/2562234151 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:19.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.790+0000 7fa7e7fff640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7e8159060 0x7fa7e81f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:19.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.791+0000 7fa7ecb0e640 1 -- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa7e814d350 msgr2=0x7fa7e8159e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:32:19.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.791+0000 7fa7ecb0e640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa7e814d350 0x7fa7e8159e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:19.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.791+0000 7fa7ecb0e640 1 -- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7e8159060 msgr2=0x7fa7e81f0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:19.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.791+0000 7fa7ecb0e640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7e8159060 0x7fa7e81f0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:19.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.791+0000 7fa7ecb0e640 1 -- 172.21.15.117:0/2562234151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e8143ad0 con 0x7fa7d0004800 2024-09-17T05:32:19.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.792+0000 7fa7ecb0e640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 0x7fa7e814b5a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006380 tx=0x7fa7d80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:19.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.792+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8015070 con 0x7fa7d0004800 2024-09-17T05:32:19.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.792+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7d8011840 con 0x7fa7d0004800 2024-09-17T05:32:19.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.793+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8011c70 con 0x7fa7d0004800 2024-09-17T05:32:19.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.793+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e80ccfe0 con 0x7fa7d0004800 2024-09-17T05:32:19.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.796+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa7d8018070 con 0x7fa7d0004800 2024-09-17T05:32:19.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.796+0000 7fa7e57fa640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa7d4078c10 0x7fa7d407b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:19.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.797+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e81f1470 con 0x7fa7d0004800 2024-09-17T05:32:19.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.797+0000 7fa7e7fff640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa7d4078c10 0x7fa7d407b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:19.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.798+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fa7d8098f70 con 0x7fa7d0004800 2024-09-17T05:32:19.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.800+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e814cd80 con 0x7fa7d0004800 2024-09-17T05:32:19.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.800+0000 7fa7e7fff640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa7d4078c10 0x7fa7d407b120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0035000 tx=0x7fa7e002d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:19.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:19.808+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa7d8062f10 con 0x7fa7d0004800 2024-09-17T05:32:20.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.286+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e80c3500 con 0x7fa7d0004800 2024-09-17T05:32:20.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.288+0000 7fa7e57fa640 1 -- 172.21.15.117:0/2562234151 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6102 (secure 0 0 0) 0x7fa7d8062810 con 0x7fa7d0004800 2024-09-17T05:32:20.290 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:20.290 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"btime":"2024-09-17T05:32:20:076895+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440,5082]},"id":1}]} 2024-09-17T05:32:20.290 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 25 2024-09-17T05:32:20.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.292+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa7d4078c10 msgr2=0x7fa7d407b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:20.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.292+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa7d4078c10 0x7fa7d407b120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0035000 tx=0x7fa7e002d040 comp rx=0 tx=0).stop 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.292+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 msgr2=0x7fa7e814b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.292+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 0x7fa7e814b5a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006380 tx=0x7fa7d80039c0 comp rx=0 tx=0).stop 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.293+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 shutdown_connections 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.293+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa7d4078c10 0x7fa7d407b120 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.293+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa7e814d350 0x7fa7e8159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.293+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7e8159060 0x7fa7e81f0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.293+0000 7fa7eed72640 1 --2- 172.21.15.117:0/2562234151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0004800 0x7fa7e814b5a0 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006380 tx=0x7fa7d80039c0 comp rx=0 tx=0).stop 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.293+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 >> 172.21.15.117:0/2562234151 conn(0x7fa7e806d190 msgr2=0x7fa7e80c50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.295+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 shutdown_connections 2024-09-17T05:32:20.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:20.295+0000 7fa7eed72640 1 -- 172.21.15.117:0/2562234151 wait complete. 2024-09-17T05:32:20.316 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=4419 2024-09-17T05:32:20.317 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.c 2024-09-17T05:32:21.663 INFO:tasks.ceph.mds.f.smithi119.stderr:2024-09-17T05:32:21.662+0000 7f6f66926640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-17T05:32:24.318 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-09-17T05:32:24.318 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:32:24.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:32:24.321 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:32:24.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.490+0000 7fc8ad142640 1 -- 172.21.15.117:0/3603894325 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00071c0 con 0x7fc894005270 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.490+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3603894325 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 msgr2=0x7fc89400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.490+0000 7fc8b03a8640 1 --2- 172.21.15.117:0/3603894325 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc89400b9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0005fd0 tx=0x7fc894005830 comp rx=0 tx=0).stop 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.491+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3603894325 shutdown_connections 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.491+0000 7fc8b03a8640 1 --2- 172.21.15.117:0/3603894325 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc89400b9c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.491+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3603894325 >> 172.21.15.117:0/3603894325 conn(0x7fc8a806d190 msgr2=0x7fc8a806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:24.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.491+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3603894325 shutdown_connections 2024-09-17T05:32:24.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.491+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3603894325 wait complete. 2024-09-17T05:32:24.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.494+0000 7fc8b03a8640 1 Processor -- start 2024-09-17T05:32:24.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.494+0000 7fc8b03a8640 1 -- start start 2024-09-17T05:32:24.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.496+0000 7fc8b03a8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc8a81edde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:24.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.497+0000 7fc8b03a8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 0x7fc8a814bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:24.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.497+0000 7fc8ad943640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 0x7fc8a814bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:24.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.497+0000 7fc8ad943640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 0x7fc8a814bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42878/0 (socket says 172.21.15.117:42878) 2024-09-17T05:32:24.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8ae144640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc8a81edde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8ae144640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc8a81edde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:51048/0 (socket says 172.21.15.117:51048) 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8ad943640 1 -- 172.21.15.117:0/3240105371 learned_addr learned my addr 172.21.15.117:0/3240105371 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8b03a8640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8a8158760 0x7fc8a8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a80c3820 con 0x7fc8a8159bb0 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a80747f0 con 0x7fc8a8158760 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8ae945640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8a8158760 0x7fc8a8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:24.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.498+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a80c67f0 con 0x7fc894005270 2024-09-17T05:32:24.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.505+0000 7fc8ad943640 1 -- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 msgr2=0x7fc8a81edde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.505+0000 7fc8ad943640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc8a81edde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.505+0000 7fc8ad943640 1 -- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8a8158760 msgr2=0x7fc8a8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.505+0000 7fc8ad943640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8a8158760 0x7fc8a8143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.505+0000 7fc8ad943640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a81ee320 con 0x7fc8a8159bb0 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.506+0000 7fc8ad943640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 0x7fc8a814bb20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc898006630 tx=0x7fc8980039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:24.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.506+0000 7fc8ae945640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8a8158760 0x7fc8a8143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:32:24.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.506+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898015070 con 0x7fc8a8159bb0 2024-09-17T05:32:24.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.507+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc898011900 con 0x7fc8a8159bb0 2024-09-17T05:32:24.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.507+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898011d30 con 0x7fc8a8159bb0 2024-09-17T05:32:24.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:32:24.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.508+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a814e760 con 0x7fc8a8159bb0 2024-09-17T05:32:24.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.509+0000 7fc8ae144640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc8a81edde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:32:24.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.509+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a814e480 con 0x7fc8a8159bb0 2024-09-17T05:32:24.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.512+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a814df10 con 0x7fc8a8159bb0 2024-09-17T05:32:24.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.516+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc898018070 con 0x7fc8a8159bb0 2024-09-17T05:32:24.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.517+0000 7fc89effd640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc88c078bc0 0x7fc88c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:24.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.517+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fc89809c200 con 0x7fc8a8159bb0 2024-09-17T05:32:24.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.518+0000 7fc8ae144640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc88c078bc0 0x7fc88c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:24.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.521+0000 7fc8ae144640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc88c078bc0 0x7fc88c07b0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc8a806c490 tx=0x7fc8a4006b70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:24.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.523+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc898064210 con 0x7fc8a8159bb0 2024-09-17T05:32:24.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.893+0000 7fc8b03a8640 1 -- 172.21.15.117:0/3240105371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8a8072e10 con 0x7fc8a8159bb0 2024-09-17T05:32:24.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.894+0000 7fc89effd640 1 -- 172.21.15.117:0/3240105371 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6102 (secure 0 0 0) 0x7fc898063b10 con 0x7fc8a8159bb0 2024-09-17T05:32:24.895 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:24.895 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":25,"btime":"2024-09-17T05:32:20:076895+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4434,4440,5082]},"id":1}]} 2024-09-17T05:32:24.895 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 25 2024-09-17T05:32:24.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.898+0000 7fc89cff9640 1 -- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc88c078bc0 msgr2=0x7fc88c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:24.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc88c078bc0 0x7fc88c07b0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc8a806c490 tx=0x7fc8a4006b70 comp rx=0 tx=0).stop 2024-09-17T05:32:24.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 -- 172.21.15.117:0/3240105371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 msgr2=0x7fc8a814bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:24.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 0x7fc8a814bb20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc898006630 tx=0x7fc8980039c0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 -- 172.21.15.117:0/3240105371 shutdown_connections 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc88c078bc0 0x7fc88c07b0d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8a8158760 0x7fc8a8143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8a8159bb0 0x7fc8a814bb20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.899+0000 7fc89cff9640 1 --2- 172.21.15.117:0/3240105371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc894005270 0x7fc8a81edde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.900+0000 7fc89cff9640 1 -- 172.21.15.117:0/3240105371 >> 172.21.15.117:0/3240105371 conn(0x7fc8a806d190 msgr2=0x7fc8a806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.900+0000 7fc89cff9640 1 -- 172.21.15.117:0/3240105371 shutdown_connections 2024-09-17T05:32:24.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:24.900+0000 7fc89cff9640 1 -- 172.21.15.117:0/3240105371 wait complete. 2024-09-17T05:32:24.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T05:32:25.311 INFO:tasks.ceph.mds.c.smithi117.stdout:starting mds.c at 2024-09-17T05:32:26.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:27.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.090+0000 7f2ff3e30640 1 -- 172.21.15.117:0/3830361795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec158a10 msgr2=0x7f2fec0c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.090+0000 7f2ff3e30640 1 --2- 172.21.15.117:0/3830361795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec158a10 0x7f2fec0c41c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0c9180 tx=0x7f2fdc02a540 comp rx=0 tx=0).stop 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.091+0000 7f2ff3e30640 1 -- 172.21.15.117:0/3830361795 shutdown_connections 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.091+0000 7f2ff3e30640 1 --2- 172.21.15.117:0/3830361795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec158a10 0x7f2fec0c41c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.091+0000 7f2ff3e30640 1 --2- 172.21.15.117:0/3830361795 >> v2:172.21.15.117:3300/0 conn(0x7f2fec152300 0x7f2fec06deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.091+0000 7f2ff3e30640 1 -- 172.21.15.117:0/3830361795 >> 172.21.15.117:0/3830361795 conn(0x7f2fec0c9a20 msgr2=0x7f2fec0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.091+0000 7f2ff3e30640 1 -- 172.21.15.117:0/3830361795 shutdown_connections 2024-09-17T05:32:27.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.091+0000 7f2ff3e30640 1 -- 172.21.15.117:0/3830361795 wait complete. 2024-09-17T05:32:27.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.094+0000 7f2ff3e30640 1 Processor -- start 2024-09-17T05:32:27.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.094+0000 7f2ff3e30640 1 -- start start 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.095+0000 7f2ff3e30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 0x7f2fec0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff3e30640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec152300 0x7f2fec149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff3e30640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fec158a10 0x7f2fec0c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff3e30640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec15b530 con 0x7f2fec0c4700 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 0x7f2fec0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 0x7f2fec0c1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42910/0 (socket says 172.21.15.117:42910) 2024-09-17T05:32:27.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 -- 172.21.15.117:0/407330909 learned_addr learned my addr 172.21.15.117:0/407330909 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:27.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff3e30640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec15c3e0 con 0x7f2fec152300 2024-09-17T05:32:27.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff3e30640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec158e20 con 0x7f2fec158a10 2024-09-17T05:32:27.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff23cd640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fec158a10 0x7f2fec0c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 -- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fec158a10 msgr2=0x7f2fec0c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fec158a10 0x7f2fec0c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 -- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec152300 msgr2=0x7f2fec149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec152300 0x7f2fec149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.096+0000 7f2ff1bcc640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec1f29b0 con 0x7f2fec0c4700 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.097+0000 7f2ff1bcc640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 0x7f2fec0c1820 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e7c0 tx=0x7f2fe800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:27.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.097+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800ccc0 con 0x7f2fec0c4700 2024-09-17T05:32:27.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.097+0000 7f2ff3e30640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fec0c0d80 con 0x7f2fec0c4700 2024-09-17T05:32:27.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.097+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fe8010430 con 0x7f2fec0c4700 2024-09-17T05:32:27.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.097+0000 7f2ff3e30640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fec1f26d0 con 0x7f2fec0c4700 2024-09-17T05:32:27.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.097+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8010860 con 0x7f2fec0c4700 2024-09-17T05:32:27.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.099+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2fe8010a00 con 0x7f2fec0c4700 2024-09-17T05:32:27.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.100+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc8005290 con 0x7f2fec0c4700 2024-09-17T05:32:27.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.101+0000 7f2fe27fc640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fd0078bc0 0x7f2fd007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:27.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.101+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f2fe809bb30 con 0x7f2fec0c4700 2024-09-17T05:32:27.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.101+0000 7f2ff13cb640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fd0078bc0 0x7f2fd007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:27.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.102+0000 7f2ff13cb640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fd0078bc0 0x7f2fd007b0d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc034000 tx=0x7f2fdc00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:27.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.108+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2fe8063580 con 0x7f2fec0c4700 2024-09-17T05:32:27.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.392+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fc8005580 con 0x7f2fec0c4700 2024-09-17T05:32:27.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.393+0000 7f2fe27fc640 1 -- 172.21.15.117:0/407330909 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7f2fe8067fb0 con 0x7f2fec0c4700 2024-09-17T05:32:27.393 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:32:27.394 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:32:27.394 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fd0078bc0 msgr2=0x7f2fd007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fd0078bc0 0x7f2fd007b0d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc034000 tx=0x7f2fdc00b040 comp rx=0 tx=0).stop 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 msgr2=0x7f2fec0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 0x7f2fec0c1820 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e7c0 tx=0x7f2fe800ec80 comp rx=0 tx=0).stop 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 shutdown_connections 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fd0078bc0 0x7f2fd007b0d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fec158a10 0x7f2fec0c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fec152300 0x7f2fec149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 --2- 172.21.15.117:0/407330909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fec0c4700 0x7f2fec0c1820 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:27.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 >> 172.21.15.117:0/407330909 conn(0x7f2fec0c9a20 msgr2=0x7f2fec06cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:27.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 shutdown_connections 2024-09-17T05:32:27.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:27.395+0000 7f2fcffff640 1 -- 172.21.15.117:0/407330909 wait complete. 2024-09-17T05:32:27.408 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-09-17T05:32:27.409 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2024-09-17T05:32:58.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:32:58.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:32:59.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:32:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.634+0000 7f93d36de640 1 -- 172.21.15.117:0/3480418532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93b80053d0 msgr2=0x7f93b800ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.634+0000 7f93d36de640 1 --2- 172.21.15.117:0/3480418532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93b80053d0 0x7f93b800ba80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f93bc001690 tx=0x7f93b8005830 comp rx=0 tx=0).stop 2024-09-17T05:32:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.634+0000 7f93d36de640 1 -- 172.21.15.117:0/3480418532 shutdown_connections 2024-09-17T05:32:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.634+0000 7f93d36de640 1 --2- 172.21.15.117:0/3480418532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93b80053d0 0x7f93b800ba80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.634+0000 7f93d36de640 1 -- 172.21.15.117:0/3480418532 >> 172.21.15.117:0/3480418532 conn(0x7f93cc06d190 msgr2=0x7f93cc06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.641+0000 7f93d36de640 1 -- 172.21.15.117:0/3480418532 shutdown_connections 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.641+0000 7f93d36de640 1 -- 172.21.15.117:0/3480418532 wait complete. 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.643+0000 7f93d36de640 1 Processor -- start 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.644+0000 7f93d36de640 1 -- start start 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.646+0000 7f93d36de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc14f460 0x7f93cc0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.646+0000 7f93d36de640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0cb8f0 0x7f93cc141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.646+0000 7f93d36de640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc14bec0 0x7f93cc14c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.646+0000 7f93d36de640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc0c5ba0 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.646+0000 7f93d36de640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc0c58c0 con 0x7f93cc14f460 2024-09-17T05:32:59.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.646+0000 7f93d36de640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f93cc0c4bd0 con 0x7f93cc14bec0 2024-09-17T05:32:59.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.647+0000 7f93d1c7b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc14bec0 0x7f93cc14c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:59.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.647+0000 7f93d1c7b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc14bec0 0x7f93cc14c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:52550/0 (socket says 172.21.15.117:52550) 2024-09-17T05:32:59.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.647+0000 7f93d1c7b640 1 -- 172.21.15.117:0/4200880264 learned_addr learned my addr 172.21.15.117:0/4200880264 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:32:59.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.647+0000 7f93d0c79640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0cb8f0 0x7f93cc141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:59.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.647+0000 7f93d147a640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc14f460 0x7f93cc0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:59.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.648+0000 7f93d0c79640 1 -- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc14bec0 msgr2=0x7f93cc14c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:59.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.648+0000 7f93d0c79640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc14bec0 0x7f93cc14c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:59.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.648+0000 7f93d0c79640 1 -- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc14f460 msgr2=0x7f93cc0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:32:59.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.648+0000 7f93d0c79640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc14f460 0x7f93cc0ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:32:59.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.648+0000 7f93d0c79640 1 -- 172.21.15.117:0/4200880264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc0ca3a0 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.652+0000 7f93d0c79640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0cb8f0 0x7f93cc141360 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f93cc072dc0 tx=0x7f93c4007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:32:59.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.652+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4016070 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.652+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93cc13bfb0 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.652+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93cc14dbc0 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.656+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93cc075170 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.655+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93c4010bd0 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.666+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4012430 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.666+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f93c4012610 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.666+0000 7f93c1ffb640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b0078c90 0x7f93b007b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:32:59.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.667+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f93c409c340 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.667+0000 7f93d147a640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b0078c90 0x7f93b007b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:32:59.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.667+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f93cc075170 con 0x7f93cc0cb8f0 2024-09-17T05:32:59.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:32:59.673+0000 7f93d147a640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b0078c90 0x7f93b007b1a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f93c8006fd0 tx=0x7f93c8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:00.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.140+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93cc0c6470 con 0x7f93cc0cb8f0 2024-09-17T05:33:00.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.142+0000 7f93c1ffb640 1 -- 172.21.15.117:0/4200880264 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7f93c4063e10 con 0x7f93cc0cb8f0 2024-09-17T05:33:00.147 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:00.147 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:00.148 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:00.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b0078c90 msgr2=0x7f93b007b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b0078c90 0x7f93b007b1a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f93c8006fd0 tx=0x7f93c8008040 comp rx=0 tx=0).stop 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0cb8f0 msgr2=0x7f93cc141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0cb8f0 0x7f93cc141360 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f93cc072dc0 tx=0x7f93c4007870 comp rx=0 tx=0).stop 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 shutdown_connections 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b0078c90 0x7f93b007b1a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc14bec0 0x7f93cc14c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0cb8f0 0x7f93cc141360 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 --2- 172.21.15.117:0/4200880264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc14f460 0x7f93cc0ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.146+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 >> 172.21.15.117:0/4200880264 conn(0x7f93cc06d190 msgr2=0x7f93cc072e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.147+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 shutdown_connections 2024-09-17T05:33:00.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.147+0000 7f93d36de640 1 -- 172.21.15.117:0/4200880264 wait complete. 2024-09-17T05:33:00.170 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=1) 2024-09-17T05:33:00.170 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T05:33:00.170 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:33:00.190 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T05:33:00.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:00.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.386+0000 7f14b91ba640 1 -- 172.21.15.117:0/3382718811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f149c004bb0 msgr2=0x7f149c005020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.386+0000 7f14b91ba640 1 --2- 172.21.15.117:0/3382718811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f149c004bb0 0x7f149c005020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00c640 tx=0x7f149c006940 comp rx=0 tx=0).stop 2024-09-17T05:33:00.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.386+0000 7f14b91ba640 1 -- 172.21.15.117:0/3382718811 shutdown_connections 2024-09-17T05:33:00.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.386+0000 7f14b91ba640 1 --2- 172.21.15.117:0/3382718811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f149c004bb0 0x7f149c005020 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.386+0000 7f14b91ba640 1 -- 172.21.15.117:0/3382718811 >> 172.21.15.117:0/3382718811 conn(0x7f14b40c9a20 msgr2=0x7f14b40c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:00.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.387+0000 7f14b91ba640 1 -- 172.21.15.117:0/3382718811 shutdown_connections 2024-09-17T05:33:00.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.388+0000 7f14b91ba640 1 -- 172.21.15.117:0/3382718811 wait complete. 2024-09-17T05:33:00.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.392+0000 7f14b91ba640 1 Processor -- start 2024-09-17T05:33:00.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.392+0000 7f14b91ba640 1 -- start start 2024-09-17T05:33:00.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b91ba640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f149c004bb0 0x7f14b414b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b91ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14b40c3180 0x7f14b4154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b91ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14b40cbd20 0x7f14b40cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b2d9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f149c004bb0 0x7f14b414b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b2d9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f149c004bb0 0x7f14b414b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:52556/0 (socket says 172.21.15.117:52556) 2024-09-17T05:33:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b2d9d640 1 -- 172.21.15.117:0/1748134888 learned_addr learned my addr 172.21.15.117:0/1748134888 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:00.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b259c640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14b40c3180 0x7f14b4154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:00.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b359e640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14b40cbd20 0x7f14b40cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:00.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.394+0000 7f14b91ba640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4157af0 con 0x7f14b40cbd20 2024-09-17T05:33:00.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b91ba640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4074370 con 0x7f14b40c3180 2024-09-17T05:33:00.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b91ba640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f14b4157dd0 con 0x7f149c004bb0 2024-09-17T05:33:00.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b359e640 1 -- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f149c004bb0 msgr2=0x7f14b414b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b359e640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f149c004bb0 0x7f14b414b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b359e640 1 -- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14b40c3180 msgr2=0x7f14b4154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.395+0000 7f14b359e640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14b40c3180 0x7f14b4154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.396+0000 7f14b359e640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b414bd60 con 0x7f14b40cbd20 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.396+0000 7f14b359e640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14b40cbd20 0x7f14b40cc130 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f14b4072b00 tx=0x7f14a80016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.397+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8022440 con 0x7f14b40cbd20 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.398+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14a80225e0 con 0x7f14b40cbd20 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.398+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a80229d0 con 0x7f14b40cbd20 2024-09-17T05:33:00.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.398+0000 7f14b91ba640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b41f0af0 con 0x7f14b40cbd20 2024-09-17T05:33:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.398+0000 7f14b91ba640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b414b020 con 0x7f14b40cbd20 2024-09-17T05:33:00.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.399+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b40c8390 con 0x7f14b40cbd20 2024-09-17T05:33:00.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.403+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f14a802b020 con 0x7f14b40cbd20 2024-09-17T05:33:00.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.404+0000 7f149b7fe640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f14a0078be0 0x7f14a007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:00.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.404+0000 7f14b2d9d640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f14a0078be0 0x7f14a007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:00.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.404+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f14a80708f0 con 0x7f14b40cbd20 2024-09-17T05:33:00.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.405+0000 7f14b2d9d640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f14a0078be0 0x7f14a007b0f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006f00 tx=0x7f14ac037040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:00.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.410+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f14a80790b0 con 0x7f14b40cbd20 2024-09-17T05:33:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.767+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14b40767b0 con 0x7f14b40cbd20 2024-09-17T05:33:00.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.768+0000 7f149b7fe640 1 -- 172.21.15.117:0/1748134888 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7f14a807df60 con 0x7f14b40cbd20 2024-09-17T05:33:00.769 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:00.769 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:00.770 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f14a0078be0 msgr2=0x7f14a007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f14a0078be0 0x7f14a007b0f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006f00 tx=0x7f14ac037040 comp rx=0 tx=0).stop 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14b40cbd20 msgr2=0x7f14b40cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14b40cbd20 0x7f14b40cc130 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f14b4072b00 tx=0x7f14a80016f0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 shutdown_connections 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f14a0078be0 0x7f14a007b0f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14b40cbd20 0x7f14b40cc130 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14b40c3180 0x7f14b4154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 --2- 172.21.15.117:0/1748134888 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f149c004bb0 0x7f14b414b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:00.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 >> 172.21.15.117:0/1748134888 conn(0x7f14b40c9a20 msgr2=0x7f14b40c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:00.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 shutdown_connections 2024-09-17T05:33:00.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:00.770+0000 7f14997fa640 1 -- 172.21.15.117:0/1748134888 wait complete. 2024-09-17T05:33:00.783 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:33:02.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:03.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.017+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/1942249710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 msgr2=0x7f4b7815ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.017+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/1942249710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b7815ae60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005fd0 tx=0x7f4b7002aa10 comp rx=0 tx=0).stop 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.018+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/1942249710 shutdown_connections 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.018+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/1942249710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b7815ae60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.018+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/1942249710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78072e10 0x7f4b781517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.018+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/1942249710 >> 172.21.15.117:0/1942249710 conn(0x7f4b7806d190 msgr2=0x7f4b7806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.018+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/1942249710 shutdown_connections 2024-09-17T05:33:03.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.018+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/1942249710 wait complete. 2024-09-17T05:33:03.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.021+0000 7f4b7f7ad640 1 Processor -- start 2024-09-17T05:33:03.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.021+0000 7f4b7f7ad640 1 -- start start 2024-09-17T05:33:03.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7f7ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b78072e10 0x7f4b78150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:03.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7f7ad640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b781f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7f7ad640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78157ea0 0x7f4b78134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7f7ad640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4b780c3820 con 0x7f4b780c6ca0 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7f7ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4b780747f0 con 0x7f4b78072e10 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7f7ad640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4b780c67f0 con 0x7f4b78157ea0 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7d549640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b78072e10 0x7f4b78150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7dd4a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78157ea0 0x7f4b78134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7dd4a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78157ea0 0x7f4b78134a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:52576/0 (socket says 172.21.15.117:52576) 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7dd4a640 1 -- 172.21.15.117:0/3223044839 learned_addr learned my addr 172.21.15.117:0/3223044839 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:03.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.023+0000 7f4b7cd48640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b781f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:03.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.024+0000 7f4b7cd48640 1 -- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78157ea0 msgr2=0x7f4b78134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:03.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.024+0000 7f4b7cd48640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78157ea0 0x7f4b78134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.024+0000 7f4b7cd48640 1 -- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b78072e10 msgr2=0x7f4b78150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:03.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.024+0000 7f4b7cd48640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b78072e10 0x7f4b78150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.024+0000 7f4b7cd48640 1 -- 172.21.15.117:0/3223044839 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7814f990 con 0x7f4b780c6ca0 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.024+0000 7f4b7cd48640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b781f2df0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400b410 tx=0x7f4b7400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.025+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74015020 con 0x7f4b780c6ca0 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.025+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b74011920 con 0x7f4b780c6ca0 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.025+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74011d70 con 0x7f4b780c6ca0 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.025+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b78150db0 con 0x7f4b780c6ca0 2024-09-17T05:33:03.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.026+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b781596e0 con 0x7f4b780c6ca0 2024-09-17T05:33:03.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.029+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b58005290 con 0x7f4b780c6ca0 2024-09-17T05:33:03.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.032+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4b74018020 con 0x7f4b780c6ca0 2024-09-17T05:33:03.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.032+0000 7f4b6dffb640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4b5c078c10 0x7f4b5c07b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:03.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.033+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f4b7405e460 con 0x7f4b780c6ca0 2024-09-17T05:33:03.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.035+0000 7f4b7d549640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4b5c078c10 0x7f4b5c07b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:03.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.042+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4b7401f070 con 0x7f4b780c6ca0 2024-09-17T05:33:03.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.053+0000 7f4b7d549640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4b5c078c10 0x7f4b5c07b120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009b30 tx=0x7f4b68005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:03.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.486+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b58005850 con 0x7f4b780c6ca0 2024-09-17T05:33:03.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.489+0000 7f4b6dffb640 1 -- 172.21.15.117:0/3223044839 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7f4b7401f070 con 0x7f4b780c6ca0 2024-09-17T05:33:03.490 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:03.491 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:03.491 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:03.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4b5c078c10 msgr2=0x7f4b5c07b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:03.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4b5c078c10 0x7f4b5c07b120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009b30 tx=0x7f4b68005f90 comp rx=0 tx=0).stop 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 msgr2=0x7f4b781f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b781f2df0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400b410 tx=0x7f4b7400b8d0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 shutdown_connections 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4b5c078c10 0x7f4b5c07b120 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b78157ea0 0x7f4b78134a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b780c6ca0 0x7f4b781f2df0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 --2- 172.21.15.117:0/3223044839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b78072e10 0x7f4b78150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.492+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 >> 172.21.15.117:0/3223044839 conn(0x7f4b7806d190 msgr2=0x7f4b781417b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:03.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.493+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 shutdown_connections 2024-09-17T05:33:03.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:03.493+0000 7f4b7f7ad640 1 -- 172.21.15.117:0/3223044839 wait complete. 2024-09-17T05:33:03.512 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:33:05.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:05.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.712+0000 7ff7e27fc640 1 -- 172.21.15.117:0/2398212153 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e40b0990 con 0x7ff7dc0d16a0 2024-09-17T05:33:05.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.713+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2398212153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 msgr2=0x7ff7dc005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:05.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.713+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2398212153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc005c00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0466c0 tx=0x7ff7e40a8820 comp rx=0 tx=0).stop 2024-09-17T05:33:05.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.713+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2398212153 shutdown_connections 2024-09-17T05:33:05.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.713+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2398212153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc005c00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:05.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.713+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2398212153 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 0x7ff7dc0db6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:05.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.713+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2398212153 >> 172.21.15.117:0/2398212153 conn(0x7ff7dc04fe30 msgr2=0x7ff7dc050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:05.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.715+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2398212153 shutdown_connections 2024-09-17T05:33:05.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.716+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2398212153 wait complete. 2024-09-17T05:33:05.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.719+0000 7ff7e8b3f640 1 Processor -- start 2024-09-17T05:33:05.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.719+0000 7ff7e8b3f640 1 -- start start 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.721+0000 7ff7e8b3f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 0x7ff7dc0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.721+0000 7ff7e8b3f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7dc0469d0 0x7ff7dc0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.721+0000 7ff7e8b3f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.721+0000 7ff7e8b3f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc045ef0 con 0x7ff7dc0d16a0 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.721+0000 7ff7e8b3f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc045c10 con 0x7ff7dc0469d0 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.721+0000 7ff7e8b3f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff7dc0461d0 con 0x7ff7dc044cb0 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e37fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 0x7ff7dc0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:05.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52430/0 (socket says 172.21.15.117:52430) 2024-09-17T05:33:05.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e3fff640 1 -- 172.21.15.117:0/2251419026 learned_addr learned my addr 172.21.15.117:0/2251419026 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:05.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e2ffd640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7dc0469d0 0x7ff7dc0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:05.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e37fe640 1 -- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7dc0469d0 msgr2=0x7ff7dc0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:05.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e37fe640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7dc0469d0 0x7ff7dc0cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:05.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e37fe640 1 -- 172.21.15.117:0/2251419026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 msgr2=0x7ff7dc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:05.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e37fe640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:05.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e37fe640 1 -- 172.21.15.117:0/2251419026 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc0ce540 con 0x7ff7dc044cb0 2024-09-17T05:33:05.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.722+0000 7ff7e3fff640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:33:05.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.723+0000 7ff7e37fe640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 0x7ff7dc0ce000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800c970 tx=0x7ff7d800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:05.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.723+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d8007bb0 con 0x7ff7dc044cb0 2024-09-17T05:33:05.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.723+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc049eb0 con 0x7ff7dc044cb0 2024-09-17T05:33:05.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.723+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7d8007d50 con 0x7ff7dc044cb0 2024-09-17T05:33:05.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.723+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d800f690 con 0x7ff7dc044cb0 2024-09-17T05:33:05.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.723+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc0d0640 con 0x7ff7dc044cb0 2024-09-17T05:33:05.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.725+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7dc0d4fc0 con 0x7ff7dc044cb0 2024-09-17T05:33:05.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.730+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff7d8018400 con 0x7ff7dc044cb0 2024-09-17T05:33:05.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.731+0000 7ff7cffff640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7c8078b90 0x7ff7c807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:05.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.731+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7ff7d809b6f0 con 0x7ff7dc044cb0 2024-09-17T05:33:05.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.732+0000 7ff7e2ffd640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7c8078b90 0x7ff7c807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:05.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.734+0000 7ff7e2ffd640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7c8078b90 0x7ff7c807b0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc050440 tx=0x7ff7d0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:05.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:05.740+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff7dc0d4fc0 con 0x7ff7dc044cb0 2024-09-17T05:33:06.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.148+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7dc04e960 con 0x7ff7dc044cb0 2024-09-17T05:33:06.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.149+0000 7ff7cffff640 1 -- 172.21.15.117:0/2251419026 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7ff7dc04e960 con 0x7ff7dc044cb0 2024-09-17T05:33:06.151 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:06.151 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:06.151 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7c8078b90 msgr2=0x7ff7c807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7c8078b90 0x7ff7c807b0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc050440 tx=0x7ff7d0009040 comp rx=0 tx=0).stop 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 msgr2=0x7ff7dc0ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 0x7ff7dc0ce000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7d800c970 tx=0x7ff7d800ce30 comp rx=0 tx=0).stop 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 shutdown_connections 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff7c8078b90 0x7ff7c807b0a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7dc0d16a0 0x7ff7dc0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7dc0469d0 0x7ff7dc0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 --2- 172.21.15.117:0/2251419026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff7dc044cb0 0x7ff7dc0ce000 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 >> 172.21.15.117:0/2251419026 conn(0x7ff7dc04fe30 msgr2=0x7ff7dc04f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:06.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 shutdown_connections 2024-09-17T05:33:06.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:06.152+0000 7ff7e8b3f640 1 -- 172.21.15.117:0/2251419026 wait complete. 2024-09-17T05:33:06.170 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:33:08.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 -- 172.21.15.117:0/3212391451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f320400de80 msgr2=0x7f32040db6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 --2- 172.21.15.117:0/3212391451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f320400de80 0x7f32040db6f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3204045270 tx=0x7f31fc02a600 comp rx=0 tx=0).stop 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 -- 172.21.15.117:0/3212391451 shutdown_connections 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 --2- 172.21.15.117:0/3212391451 >> v2:172.21.15.119:3300/0 conn(0x7f3204046a00 0x7f32040d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 --2- 172.21.15.117:0/3212391451 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3204044ce0 0x7f3204005c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 --2- 172.21.15.117:0/3212391451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f320400de80 0x7f32040db6f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 -- 172.21.15.117:0/3212391451 >> 172.21.15.117:0/3212391451 conn(0x7f320404fe60 msgr2=0x7f3204050270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:08.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 -- 172.21.15.117:0/3212391451 shutdown_connections 2024-09-17T05:33:08.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.368+0000 7f32114d5640 1 -- 172.21.15.117:0/3212391451 wait complete. 2024-09-17T05:33:08.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.371+0000 7f32114d5640 1 Processor -- start 2024-09-17T05:33:08.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.371+0000 7f32114d5640 1 -- start start 2024-09-17T05:33:08.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.373+0000 7f32114d5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3204044ce0 0x7f32040c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:08.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.373+0000 7f32114d5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3204046a00 0x7f3204005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:08.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.373+0000 7f32114d5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f32040d82d0 0x7f32040c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:08.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.373+0000 7f32114d5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f32040d1a30 con 0x7f3204044ce0 2024-09-17T05:33:08.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.373+0000 7f32114d5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3204046200 con 0x7f3204046a00 2024-09-17T05:33:08.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.373+0000 7f32114d5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f32040d5230 con 0x7f32040d82d0 2024-09-17T05:33:08.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f32040d82d0 0x7f32040c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:08.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320b7fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f32040d82d0 0x7f32040c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:43120/0 (socket says 172.21.15.117:43120) 2024-09-17T05:33:08.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320b7fe640 1 -- 172.21.15.117:0/2180437775 learned_addr learned my addr 172.21.15.117:0/2180437775 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:08.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320affd640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3204044ce0 0x7f32040c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:08.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320a7fc640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3204046a00 0x7f3204005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:08.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320affd640 1 -- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f32040d82d0 msgr2=0x7f32040c3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:08.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320affd640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f32040d82d0 0x7f32040c3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320affd640 1 -- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3204046a00 msgr2=0x7f3204005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:08.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320affd640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3204046a00 0x7f3204005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.374+0000 7f320affd640 1 -- 172.21.15.117:0/2180437775 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32040c8bf0 con 0x7f3204044ce0 2024-09-17T05:33:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.375+0000 7f320affd640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3204044ce0 0x7f32040c86b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31fc004000 tx=0x7f31fc004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.376+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc041070 con 0x7f3204044ce0 2024-09-17T05:33:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.376+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31fc005220 con 0x7f3204044ce0 2024-09-17T05:33:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.376+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc041070 con 0x7f3204044ce0 2024-09-17T05:33:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.376+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320417d700 con 0x7f3204044ce0 2024-09-17T05:33:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.376+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32040c7bd0 con 0x7f3204044ce0 2024-09-17T05:33:08.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.378+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32040db6f0 con 0x7f3204044ce0 2024-09-17T05:33:08.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.379+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f31fc0053d0 con 0x7f3204044ce0 2024-09-17T05:33:08.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.380+0000 7f31f37fe640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f31f8078fc0 0x7f31f807b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:08.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.381+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f31fc005020 con 0x7f3204044ce0 2024-09-17T05:33:08.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.381+0000 7f320a7fc640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f31f8078fc0 0x7f31f807b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:08.383 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.382+0000 7f320a7fc640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f31f8078fc0 0x7f31f807b4d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f320404a540 tx=0x7f320000b260 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:08.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.388+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f31fc02ae20 con 0x7f3204044ce0 2024-09-17T05:33:08.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.834+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3204032a40 con 0x7f3204044ce0 2024-09-17T05:33:08.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.836+0000 7f31f37fe640 1 -- 172.21.15.117:0/2180437775 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7f31fc083180 con 0x7f3204044ce0 2024-09-17T05:33:08.838 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:08.838 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:08.838 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:08.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f31f8078fc0 msgr2=0x7f31f807b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:08.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f31f8078fc0 0x7f31f807b4d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f320404a540 tx=0x7f320000b260 comp rx=0 tx=0).stop 2024-09-17T05:33:08.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3204044ce0 msgr2=0x7f32040c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:08.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3204044ce0 0x7f32040c86b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31fc004000 tx=0x7f31fc004240 comp rx=0 tx=0).stop 2024-09-17T05:33:08.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 shutdown_connections 2024-09-17T05:33:08.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f31f8078fc0 0x7f31f807b4d0 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f320404a540 tx=0x7f320000b260 comp rx=0 tx=0).stop 2024-09-17T05:33:08.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f32040d82d0 0x7f32040c3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3204046a00 0x7f3204005bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 --2- 172.21.15.117:0/2180437775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3204044ce0 0x7f32040c86b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:08.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.841+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 >> 172.21.15.117:0/2180437775 conn(0x7f320404fe60 msgr2=0x7f3204006c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:08.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.844+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 shutdown_connections 2024-09-17T05:33:08.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:08.844+0000 7f32114d5640 1 -- 172.21.15.117:0/2180437775 wait complete. 2024-09-17T05:33:08.867 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:33:10.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.089+0000 7f8d916d3640 1 -- 172.21.15.117:0/2916550953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 msgr2=0x7f8d8c173910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:11.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.089+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2916550953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c173910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0c01b0 tx=0x7f8d8402a7c0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.090+0000 7f8d916d3640 1 -- 172.21.15.117:0/2916550953 shutdown_connections 2024-09-17T05:33:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.090+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2916550953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c173910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.090+0000 7f8d916d3640 1 -- 172.21.15.117:0/2916550953 >> 172.21.15.117:0/2916550953 conn(0x7f8d8c06d190 msgr2=0x7f8d8c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:11.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.096+0000 7f8d916d3640 1 -- 172.21.15.117:0/2916550953 shutdown_connections 2024-09-17T05:33:11.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.097+0000 7f8d916d3640 1 -- 172.21.15.117:0/2916550953 wait complete. 2024-09-17T05:33:11.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.100+0000 7f8d916d3640 1 Processor -- start 2024-09-17T05:33:11.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.103+0000 7f8d916d3640 1 -- start start 2024-09-17T05:33:11.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d916d3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c0cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:11.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d916d3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d8c08c4c0 0x7f8d8c0c2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d916d3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c0c62d0 0x7f8d8c1648e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d916d3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c16b5d0 con 0x7f8d8c074e60 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d916d3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c0c66e0 con 0x7f8d8c0c62d0 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d916d3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8c0c8b80 con 0x7f8d8c08c4c0 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d8affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c0cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d8affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c0cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52462/0 (socket says 172.21.15.117:52462) 2024-09-17T05:33:11.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.105+0000 7f8d8affd640 1 -- 172.21.15.117:0/2733704264 learned_addr learned my addr 172.21.15.117:0/2733704264 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:11.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.106+0000 7f8d8affd640 1 -- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d8c08c4c0 msgr2=0x7f8d8c0c2460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:33:11.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.106+0000 7f8d8affd640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d8c08c4c0 0x7f8d8c0c2460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.106+0000 7f8d8affd640 1 -- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c0c62d0 msgr2=0x7f8d8c1648e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:33:11.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.106+0000 7f8d8affd640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c0c62d0 0x7f8d8c1648e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.106+0000 7f8d8affd640 1 -- 172.21.15.117:0/2733704264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1656f0 con 0x7f8d8c074e60 2024-09-17T05:33:11.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.106+0000 7f8d8affd640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c0cce60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00b410 tx=0x7f8d7c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:11.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.107+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c015020 con 0x7f8d8c074e60 2024-09-17T05:33:11.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.107+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d7c011920 con 0x7f8d8c074e60 2024-09-17T05:33:11.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.107+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c0c29a0 con 0x7f8d8c074e60 2024-09-17T05:33:11.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.107+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c165410 con 0x7f8d8c074e60 2024-09-17T05:33:11.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.108+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c011d30 con 0x7f8d8c074e60 2024-09-17T05:33:11.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.112+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d68005290 con 0x7f8d8c074e60 2024-09-17T05:33:11.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.122+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8d7c018020 con 0x7f8d8c074e60 2024-09-17T05:33:11.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.123+0000 7f8d7b7fe640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8d70078c60 0x7f8d7007b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:11.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.124+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7f8d7c09b0c0 con 0x7f8d8c074e60 2024-09-17T05:33:11.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.124+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8d7c01f020 con 0x7f8d8c074e60 2024-09-17T05:33:11.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.124+0000 7f8d8a7fc640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8d70078c60 0x7f8d7007b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:11.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.131+0000 7f8d8a7fc640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8d70078c60 0x7f8d7007b170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0cca80 tx=0x7f8d80007a10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:11.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.624+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d68005580 con 0x7f8d8c074e60 2024-09-17T05:33:11.630 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:11.630 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:11.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.626+0000 7f8d7b7fe640 1 -- 172.21.15.117:0/2733704264 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7f8d8c0c29a0 con 0x7f8d8c074e60 2024-09-17T05:33:11.631 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8d70078c60 msgr2=0x7f8d7007b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8d70078c60 0x7f8d7007b170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0cca80 tx=0x7f8d80007a10 comp rx=0 tx=0).stop 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 msgr2=0x7f8d8c0cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c0cce60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00b410 tx=0x7f8d7c00b8d0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 shutdown_connections 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8d70078c60 0x7f8d7007b170 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c0c62d0 0x7f8d8c1648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d8c08c4c0 0x7f8d8c0c2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 --2- 172.21.15.117:0/2733704264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d8c074e60 0x7f8d8c0cce60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 >> 172.21.15.117:0/2733704264 conn(0x7f8d8c06d190 msgr2=0x7f8d8c166d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.634+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 shutdown_connections 2024-09-17T05:33:11.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:11.635+0000 7f8d916d3640 1 -- 172.21.15.117:0/2733704264 wait complete. 2024-09-17T05:33:11.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:33:13.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:13.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.884+0000 7fdb8e7fc640 1 -- 172.21.15.117:0/2186841723 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c031600 con 0x7fdb8804e7b0 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.885+0000 7fdb948be640 1 -- 172.21.15.117:0/2186841723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb8804e7b0 msgr2=0x7fdb8804ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.885+0000 7fdb948be640 1 --2- 172.21.15.117:0/2186841723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb8804e7b0 0x7fdb8804ebc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009830 tx=0x7fdb7c02a5b0 comp rx=0 tx=0).stop 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.885+0000 7fdb948be640 1 -- 172.21.15.117:0/2186841723 shutdown_connections 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.885+0000 7fdb948be640 1 --2- 172.21.15.117:0/2186841723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb8804e7b0 0x7fdb8804ebc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.885+0000 7fdb948be640 1 --2- 172.21.15.117:0/2186841723 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804d470 0x7fdb88052510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.885+0000 7fdb948be640 1 -- 172.21.15.117:0/2186841723 >> 172.21.15.117:0/2186841723 conn(0x7fdb88009d20 msgr2=0x7fdb8800a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:13.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.886+0000 7fdb948be640 1 -- 172.21.15.117:0/2186841723 shutdown_connections 2024-09-17T05:33:13.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.888+0000 7fdb948be640 1 -- 172.21.15.117:0/2186841723 wait complete. 2024-09-17T05:33:13.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.891+0000 7fdb948be640 1 Processor -- start 2024-09-17T05:33:13.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.891+0000 7fdb948be640 1 -- start start 2024-09-17T05:33:13.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.893+0000 7fdb948be640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8804d470 0x7fdb880d58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:13.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb948be640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 0x7fdb880d8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:13.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb948be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb880d54b0 0x7fdb880cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:13.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb8f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8804d470 0x7fdb880d58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:13.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb948be640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdb880dec30 con 0x7fdb880d54b0 2024-09-17T05:33:13.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb948be640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8804f100 con 0x7fdb8804d470 2024-09-17T05:33:13.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb8effd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 0x7fdb880d8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:13.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb948be640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fdb880097f0 con 0x7fdb8804e7b0 2024-09-17T05:33:13.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb8effd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 0x7fdb880d8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:43144/0 (socket says 172.21.15.117:43144) 2024-09-17T05:33:13.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb8effd640 1 -- 172.21.15.117:0/2974904933 learned_addr learned my addr 172.21.15.117:0/2974904933 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:13.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.896+0000 7fdb8ffff640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb880d54b0 0x7fdb880cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:13.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8effd640 1 -- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8804d470 msgr2=0x7fdb880d58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8effd640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8804d470 0x7fdb880d58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8effd640 1 -- 172.21.15.117:0/2974904933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb880d54b0 msgr2=0x7fdb880cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8effd640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb880d54b0 0x7fdb880cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8effd640 1 -- 172.21.15.117:0/2974904933 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb880e3390 con 0x7fdb8804e7b0 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8ffff640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb880d54b0 0x7fdb880cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8f7fe640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8804d470 0x7fdb880d58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:33:13.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.899+0000 7fdb8effd640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 0x7fdb880d8130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c02aba0 tx=0x7fdb7c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:13.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.900+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c005c90 con 0x7fdb8804e7b0 2024-09-17T05:33:13.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.900+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb7c031d60 con 0x7fdb8804e7b0 2024-09-17T05:33:13.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.900+0000 7fdb948be640 1 -- 172.21.15.117:0/2974904933 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb8804bf80 con 0x7fdb8804e7b0 2024-09-17T05:33:13.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.900+0000 7fdb948be640 1 -- 172.21.15.117:0/2974904933 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb880e30b0 con 0x7fdb8804e7b0 2024-09-17T05:33:13.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.900+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c033330 con 0x7fdb8804e7b0 2024-09-17T05:33:13.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.904+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdb7c0334d0 con 0x7fdb8804e7b0 2024-09-17T05:33:13.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.905+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb8804e040 con 0x7fdb8804e7b0 2024-09-17T05:33:13.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.906+0000 7fdb7bfff640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdb74078c90 0x7fdb7407b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:13.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.906+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6762+0+0 (secure 0 0 0) 0x7fdb7c0bc140 con 0x7fdb8804e7b0 2024-09-17T05:33:13.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.911+0000 7fdb8f7fe640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdb74078c90 0x7fdb7407b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:13.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.914+0000 7fdb8f7fe640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdb74078c90 0x7fdb7407b1a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdb900524b0 tx=0x7fdb9008f040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:13.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:13.916+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdb7c03f030 con 0x7fdb8804e7b0 2024-09-17T05:33:14.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.364+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb880cfea0 con 0x7fdb8804e7b0 2024-09-17T05:33:14.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.366+0000 7fdb7bfff640 1 -- 172.21.15.117:0/2974904933 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6927 (secure 0 0 0) 0x7fdb7c031770 con 0x7fdb8804e7b0 2024-09-17T05:33:14.367 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:14.367 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":26,"btime":"2024-09-17T05:32:25:508880+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:32:20.076886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":4434,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4434":{"gid":4434,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2224210538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2224210538},{"type":"v1","addr":"172.21.15.119:6835","nonce":2224210538}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440,4434]},"id":1}]} 2024-09-17T05:33:14.367 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 26 2024-09-17T05:33:14.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdb74078c90 msgr2=0x7fdb7407b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:14.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdb74078c90 0x7fdb7407b1a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdb900524b0 tx=0x7fdb9008f040 comp rx=0 tx=0).stop 2024-09-17T05:33:14.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 msgr2=0x7fdb880d8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:14.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 0x7fdb880d8130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c02aba0 tx=0x7fdb7c033c20 comp rx=0 tx=0).stop 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 shutdown_connections 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdb74078c90 0x7fdb7407b1a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb880d54b0 0x7fdb880cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdb8804e7b0 0x7fdb880d8130 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 --2- 172.21.15.117:0/2974904933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8804d470 0x7fdb880d58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 >> 172.21.15.117:0/2974904933 conn(0x7fdb88009d20 msgr2=0x7fdb880de1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 shutdown_connections 2024-09-17T05:33:14.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:14.371+0000 7fdb79ffb640 1 -- 172.21.15.117:0/2974904933 wait complete. 2024-09-17T05:33:14.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:33:16.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.594+0000 7f032e66c640 1 -- 172.21.15.117:0/665952623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03280cc910 msgr2=0x7f03280c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.594+0000 7f032e66c640 1 --2- 172.21.15.117:0/665952623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03280cc910 0x7f03280c7890 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0324009830 tx=0x7f032402a5d0 comp rx=0 tx=0).stop 2024-09-17T05:33:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.595+0000 7f032e66c640 1 -- 172.21.15.117:0/665952623 shutdown_connections 2024-09-17T05:33:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.595+0000 7f032e66c640 1 --2- 172.21.15.117:0/665952623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03280cc910 0x7f03280c7890 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:16.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.595+0000 7f032e66c640 1 --2- 172.21.15.117:0/665952623 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03281516d0 0x7f0328157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.595+0000 7f032e66c640 1 -- 172.21.15.117:0/665952623 >> 172.21.15.117:0/665952623 conn(0x7f032806cda0 msgr2=0x7f032806d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.595+0000 7f032e66c640 1 -- 172.21.15.117:0/665952623 shutdown_connections 2024-09-17T05:33:16.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.596+0000 7f032e66c640 1 -- 172.21.15.117:0/665952623 wait complete. 2024-09-17T05:33:16.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.599+0000 7f032e66c640 1 Processor -- start 2024-09-17T05:33:16.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.599+0000 7f032e66c640 1 -- start start 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032e66c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03280c7dd0 0x7f03281531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032e66c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 0x7f032815ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032e66c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0328159c10 0x7f032806fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032e66c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0328158330 con 0x7f03281516d0 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032e66c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f03281575f0 con 0x7f0328159c10 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032e66c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f03280cbc80 con 0x7f03280c7dd0 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032ce69640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 0x7f032815ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032ce69640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 0x7f032815ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42278/0 (socket says 172.21.15.117:42278) 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.601+0000 7f032ce69640 1 -- 172.21.15.117:0/3104559788 learned_addr learned my addr 172.21.15.117:0/3104559788 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:16.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032d66a640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03280c7dd0 0x7f03281531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032de6b640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0328159c10 0x7f032806fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032ce69640 1 -- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03280c7dd0 msgr2=0x7f03281531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032ce69640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03280c7dd0 0x7f03281531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:16.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032ce69640 1 -- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0328159c10 msgr2=0x7f032806fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032ce69640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0328159c10 0x7f032806fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032ce69640 1 -- 172.21.15.117:0/3104559788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328158e50 con 0x7f03281516d0 2024-09-17T05:33:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.602+0000 7f032ce69640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 0x7f032815ab90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f032402ab40 tx=0x7f0324032c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.603+0000 7f032d66a640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03280c7dd0 0x7f03281531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:33:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.603+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324040070 con 0x7f03281516d0 2024-09-17T05:33:16.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.603+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0328159130 con 0x7f03281516d0 2024-09-17T05:33:16.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.603+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03280c3200 con 0x7f03281516d0 2024-09-17T05:33:16.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.607+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0324003800 con 0x7f03281516d0 2024-09-17T05:33:16.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.607+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324031600 con 0x7f03281516d0 2024-09-17T05:33:16.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.607+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0328076550 con 0x7f03281516d0 2024-09-17T05:33:16.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.612+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0324039040 con 0x7f03281516d0 2024-09-17T05:33:16.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.613+0000 7f031dffb640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f030c078c60 0x7f030c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:16.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.613+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f03240bc510 con 0x7f03281516d0 2024-09-17T05:33:16.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.614+0000 7f032d66a640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f030c078c60 0x7f030c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:16.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.617+0000 7f032d66a640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f030c078c60 0x7f030c07b170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0318009aa0 tx=0x7f0318005db0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:16.618 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:16.617+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0328076550 con 0x7f03281516d0 2024-09-17T05:33:17.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.067+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03280c93d0 con 0x7f03281516d0 2024-09-17T05:33:17.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.068+0000 7f031dffb640 1 -- 172.21.15.117:0/3104559788 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7f0324084600 con 0x7f03281516d0 2024-09-17T05:33:17.069 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:17.070 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":27,"btime":"2024-09-17T05:33:15:876298+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:15.876293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:17.070 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 27 2024-09-17T05:33:17.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.073+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f030c078c60 msgr2=0x7f030c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.073+0000 7f032e66c640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f030c078c60 0x7f030c07b170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0318009aa0 tx=0x7f0318005db0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.073+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 msgr2=0x7f032815ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.073+0000 7f032e66c640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 0x7f032815ab90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f032402ab40 tx=0x7f0324032c50 comp rx=0 tx=0).stop 2024-09-17T05:33:17.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.075+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 shutdown_connections 2024-09-17T05:33:17.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f030c078c60 0x7f030c07b170 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0328159c10 0x7f032806fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03281516d0 0x7f032815ab90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 --2- 172.21.15.117:0/3104559788 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03280c7dd0 0x7f03281531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 >> 172.21.15.117:0/3104559788 conn(0x7f032806cda0 msgr2=0x7f032806e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:17.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 shutdown_connections 2024-09-17T05:33:17.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.076+0000 7f032e66c640 1 -- 172.21.15.117:0/3104559788 wait complete. 2024-09-17T05:33:17.101 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-09-17T05:33:17.101 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-09-17T05:33:17.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:17.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.303+0000 7f6ff7fff640 1 -- 172.21.15.117:0/999243897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff00469d0 msgr2=0x7f6ff00db6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.303+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/999243897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff00469d0 0x7f6ff00db6c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0034260 tx=0x7f6fe802b230 comp rx=0 tx=0).stop 2024-09-17T05:33:17.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.303+0000 7f6ff7fff640 1 -- 172.21.15.117:0/999243897 shutdown_connections 2024-09-17T05:33:17.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.303+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/999243897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff00469d0 0x7f6ff00db6c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.303+0000 7f6ff7fff640 1 -- 172.21.15.117:0/999243897 >> 172.21.15.117:0/999243897 conn(0x7f6ff004fe30 msgr2=0x7f6ff0050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:17.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.306+0000 7f6ff7fff640 1 -- 172.21.15.117:0/999243897 shutdown_connections 2024-09-17T05:33:17.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.306+0000 7f6ff7fff640 1 -- 172.21.15.117:0/999243897 wait complete. 2024-09-17T05:33:17.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.309+0000 7f6ff7fff640 1 Processor -- start 2024-09-17T05:33:17.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.309+0000 7f6ff7fff640 1 -- start start 2024-09-17T05:33:17.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.311+0000 7f6ff7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff0044cb0 0x7f6ff00ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:17.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.311+0000 7f6ff7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ff00469d0 0x7f6ff0034bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:17.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.311+0000 7f6ff7fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ff00d16a0 0x7f6ff00cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:17.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.311+0000 7f6ff7fff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff0045ef0 con 0x7f6ff0044cb0 2024-09-17T05:33:17.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.311+0000 7f6ff7fff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff0045c10 con 0x7f6ff00469d0 2024-09-17T05:33:17.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.311+0000 7f6ff7fff640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff00461d0 con 0x7f6ff00d16a0 2024-09-17T05:33:17.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ff00d16a0 0x7f6ff00cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:17.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ff00d16a0 0x7f6ff00cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:57076/0 (socket says 172.21.15.117:57076) 2024-09-17T05:33:17.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff77fe640 1 -- 172.21.15.117:0/4173973121 learned_addr learned my addr 172.21.15.117:0/4173973121 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:17.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff67fc640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ff00469d0 0x7f6ff0034bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:17.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff6ffd640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff0044cb0 0x7f6ff00ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:17.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff67fc640 1 -- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ff00d16a0 msgr2=0x7f6ff00cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff67fc640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ff00d16a0 0x7f6ff00cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff67fc640 1 -- 172.21.15.117:0/4173973121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff0044cb0 msgr2=0x7f6ff00ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff67fc640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff0044cb0 0x7f6ff00ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.312+0000 7f6ff67fc640 1 -- 172.21.15.117:0/4173973121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff00cd030 con 0x7f6ff00469d0 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.313+0000 7f6ff6ffd640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff0044cb0 0x7f6ff00ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.313+0000 7f6ff67fc640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ff00469d0 0x7f6ff0034bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8031000 tx=0x7f6fe80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.313+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8041020 con 0x7f6ff00469d0 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.314+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fe8003c20 con 0x7f6ff00469d0 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.314+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe800a330 con 0x7f6ff00469d0 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.314+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff00cc610 con 0x7f6ff00469d0 2024-09-17T05:33:17.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.314+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff00d4270 con 0x7f6ff00469d0 2024-09-17T05:33:17.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.316+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff000de50 con 0x7f6ff00469d0 2024-09-17T05:33:17.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.316+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6fe8044020 con 0x7f6ff00469d0 2024-09-17T05:33:17.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.317+0000 7f6fdf7fe640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6fe4078b90 0x7f6fe407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:17.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.317+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f6fe80bbab0 con 0x7f6ff00469d0 2024-09-17T05:33:17.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.318+0000 7f6ff6ffd640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6fe4078b90 0x7f6fe407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:17.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.321+0000 7f6ff6ffd640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6fe4078b90 0x7f6fe407b0a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009d50 tx=0x7f6fec009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:17.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.326+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6fe804b030 con 0x7f6ff00469d0 2024-09-17T05:33:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.789+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ff0032a10 con 0x7f6ff00469d0 2024-09-17T05:33:17.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.794+0000 7f6fdf7fe640 1 -- 172.21.15.117:0/4173973121 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7f6fe8083a10 con 0x7f6ff00469d0 2024-09-17T05:33:17.796 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:17.796 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":27,"btime":"2024-09-17T05:33:15:876298+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:15.876293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:17.796 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 27 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6fe4078b90 msgr2=0x7f6fe407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6fe4078b90 0x7f6fe407b0a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009d50 tx=0x7f6fec009060 comp rx=0 tx=0).stop 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ff00469d0 msgr2=0x7f6ff0034bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ff00469d0 0x7f6ff0034bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8031000 tx=0x7f6fe80045e0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 shutdown_connections 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6fe4078b90 0x7f6fe407b0a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ff00d16a0 0x7f6ff00cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ff00469d0 0x7f6ff0034bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 --2- 172.21.15.117:0/4173973121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ff0044cb0 0x7f6ff00ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:17.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 >> 172.21.15.117:0/4173973121 conn(0x7f6ff004fe30 msgr2=0x7f6ff004f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:17.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 shutdown_connections 2024-09-17T05:33:17.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:17.798+0000 7f6ff7fff640 1 -- 172.21.15.117:0/4173973121 wait complete. 2024-09-17T05:33:17.819 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:33:15:876298+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 46 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:33:15.876293+0000", "qdb_cluster": [ 4440, 5082 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:33:19.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:20.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.013+0000 7feac73c5640 1 -- 172.21.15.117:0/1044883617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac00cd670 msgr2=0x7feac00cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:20.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.013+0000 7feac73c5640 1 --2- 172.21.15.117:0/1044883617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac00cd670 0x7feac00cda80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feab8005fd0 tx=0x7feab802a8b0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.014+0000 7feac73c5640 1 -- 172.21.15.117:0/1044883617 shutdown_connections 2024-09-17T05:33:20.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.014+0000 7feac73c5640 1 --2- 172.21.15.117:0/1044883617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac00cd670 0x7feac00cda80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.014+0000 7feac73c5640 1 -- 172.21.15.117:0/1044883617 >> 172.21.15.117:0/1044883617 conn(0x7feac0076430 msgr2=0x7feac006dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:20.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.019+0000 7feac73c5640 1 -- 172.21.15.117:0/1044883617 shutdown_connections 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.022+0000 7feac73c5640 1 -- 172.21.15.117:0/1044883617 wait complete. 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.025+0000 7feac73c5640 1 Processor -- start 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.025+0000 7feac73c5640 1 -- start start 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.027+0000 7feac73c5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 0x7feac015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.027+0000 7feac73c5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feac00ccc60 0x7feac006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.027+0000 7feac73c5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac0071820 0x7feac020efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:20.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.027+0000 7feac73c5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feac015f990 con 0x7feac00b5ac0 2024-09-17T05:33:20.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.027+0000 7feac73c5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feac015f6b0 con 0x7feac0071820 2024-09-17T05:33:20.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.027+0000 7feac73c5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7feac015fc70 con 0x7feac00ccc60 2024-09-17T05:33:20.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 0x7feac015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:20.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 0x7feac015c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42296/0 (socket says 172.21.15.117:42296) 2024-09-17T05:33:20.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 -- 172.21.15.117:0/1124301063 learned_addr learned my addr 172.21.15.117:0/1124301063 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:20.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 -- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feac00ccc60 msgr2=0x7feac006fa50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:33:20.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feac00ccc60 0x7feac006fa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 -- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac0071820 msgr2=0x7feac020efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:20.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac0071820 0x7feac020efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.029+0000 7feac63c3640 1 -- 172.21.15.117:0/1124301063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac006ff90 con 0x7feac00b5ac0 2024-09-17T05:33:20.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.030+0000 7feac63c3640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 0x7feac015c7a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c430 tx=0x7feabc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:20.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.031+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc011850 con 0x7feac00b5ac0 2024-09-17T05:33:20.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.031+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac020fa10 con 0x7feac00b5ac0 2024-09-17T05:33:20.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.031+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac0162f10 con 0x7feac00b5ac0 2024-09-17T05:33:20.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.031+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feabc017070 con 0x7feac00b5ac0 2024-09-17T05:33:20.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.031+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc011da0 con 0x7feac00b5ac0 2024-09-17T05:33:20.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.035+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feabc01c460 con 0x7feac00b5ac0 2024-09-17T05:33:20.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.037+0000 7feab6ffd640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feaa4078c60 0x7feaa407b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:20.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.037+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7feabc09ae40 con 0x7feac00b5ac0 2024-09-17T05:33:20.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.038+0000 7feac5bc2640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feaa4078c60 0x7feaa407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:20.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.038+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa8005290 con 0x7feac00b5ac0 2024-09-17T05:33:20.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.047+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feabc062ea0 con 0x7feac00b5ac0 2024-09-17T05:33:20.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.050+0000 7feac5bc2640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feaa4078c60 0x7feaa407b170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feac0160ab0 tx=0x7feab0006b30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:20.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.530+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feaa8005580 con 0x7feac00b5ac0 2024-09-17T05:33:20.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.534+0000 7feab6ffd640 1 -- 172.21.15.117:0/1124301063 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7feabc0627a0 con 0x7feac00b5ac0 2024-09-17T05:33:20.539 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:20.539 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":27,"btime":"2024-09-17T05:33:15:876298+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:15.876293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:20.539 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 27 2024-09-17T05:33:20.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.540+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feaa4078c60 msgr2=0x7feaa407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:20.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.540+0000 7feac73c5640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feaa4078c60 0x7feaa407b170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feac0160ab0 tx=0x7feab0006b30 comp rx=0 tx=0).stop 2024-09-17T05:33:20.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.540+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 msgr2=0x7feac015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:20.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.540+0000 7feac73c5640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 0x7feac015c7a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c430 tx=0x7feabc00c8f0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 shutdown_connections 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7feaa4078c60 0x7feaa407b170 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac0071820 0x7feac020efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7feac00ccc60 0x7feac006fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 --2- 172.21.15.117:0/1124301063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feac00b5ac0 0x7feac015c7a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 >> 172.21.15.117:0/1124301063 conn(0x7feac0076430 msgr2=0x7feac00cb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 shutdown_connections 2024-09-17T05:33:20.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:20.541+0000 7feac73c5640 1 -- 172.21.15.117:0/1124301063 wait complete. 2024-09-17T05:33:20.561 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:22.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:22.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.776+0000 7ff89f59e640 1 -- 172.21.15.117:0/1970731355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a0074c10 msgr2=0x7ff8a0074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:22.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.776+0000 7ff89f59e640 1 --2- 172.21.15.117:0/1970731355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a0074c10 0x7ff8a0074340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff898005fd0 tx=0x7ff89802ad20 comp rx=0 tx=0).stop 2024-09-17T05:33:22.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.779+0000 7ff89f59e640 1 -- 172.21.15.117:0/1970731355 shutdown_connections 2024-09-17T05:33:22.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.779+0000 7ff89f59e640 1 --2- 172.21.15.117:0/1970731355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a0074c10 0x7ff8a0074340 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:22.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.779+0000 7ff89f59e640 1 -- 172.21.15.117:0/1970731355 >> 172.21.15.117:0/1970731355 conn(0x7ff8a00ca0c0 msgr2=0x7ff8a00ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:22.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.779+0000 7ff89f59e640 1 -- 172.21.15.117:0/1970731355 shutdown_connections 2024-09-17T05:33:22.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.779+0000 7ff89f59e640 1 -- 172.21.15.117:0/1970731355 wait complete. 2024-09-17T05:33:22.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.783+0000 7ff89f59e640 1 Processor -- start 2024-09-17T05:33:22.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.783+0000 7ff89f59e640 1 -- start start 2024-09-17T05:33:22.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.785+0000 7ff89f59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a006cda0 0x7ff8a01f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:22.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89f59e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 0x7ff8a0152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89f59e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8a00c5910 0x7ff8a01ea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89f59e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a015ff10 con 0x7ff8a006cda0 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89f59e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a0160280 con 0x7ff8a006d8c0 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89f59e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a00c92b0 con 0x7ff8a00c5910 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 0x7ff8a0152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 0x7ff8a0152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:56764/0 (socket says 172.21.15.117:56764) 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.787+0000 7ff89dd9b640 1 -- 172.21.15.117:0/3477541534 learned_addr learned my addr 172.21.15.117:0/3477541534 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.788+0000 7ff89ed9d640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8a00c5910 0x7ff8a01ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89dd9b640 1 -- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8a00c5910 msgr2=0x7ff8a01ea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89dd9b640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8a00c5910 0x7ff8a01ea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:22.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89dd9b640 1 -- 172.21.15.117:0/3477541534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a006cda0 msgr2=0x7ff8a01f61a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:33:22.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89e59c640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a006cda0 0x7ff8a01f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:22.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89dd9b640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a006cda0 0x7ff8a01f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:22.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89dd9b640 1 -- 172.21.15.117:0/3477541534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a00cb880 con 0x7ff8a006d8c0 2024-09-17T05:33:22.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.789+0000 7ff89e59c640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a006cda0 0x7ff8a01f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:33:22.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.790+0000 7ff89dd9b640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 0x7ff8a0152540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff898009e50 tx=0x7ff898003f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:22.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.793+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898038040 con 0x7ff8a006d8c0 2024-09-17T05:33:22.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.793+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8980044f0 con 0x7ff8a006d8c0 2024-09-17T05:33:22.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.793+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a01f66e0 con 0x7ff8a006d8c0 2024-09-17T05:33:22.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.793+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898033440 con 0x7ff8a006d8c0 2024-09-17T05:33:22.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.793+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a01f59a0 con 0x7ff8a006d8c0 2024-09-17T05:33:22.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.796+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff8980335e0 con 0x7ff8a006d8c0 2024-09-17T05:33:22.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.797+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a006e770 con 0x7ff8a006d8c0 2024-09-17T05:33:22.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.797+0000 7ff88effd640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff880078be0 0x7ff88007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:22.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.798+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7ff8980bc260 con 0x7ff8a006d8c0 2024-09-17T05:33:22.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.800+0000 7ff89e59c640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff880078be0 0x7ff88007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:22.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.801+0000 7ff89e59c640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff880078be0 0x7ff88007b0f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff89000a830 tx=0x7ff890008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:22.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:22.811+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff8a006e770 con 0x7ff8a006d8c0 2024-09-17T05:33:23.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.283+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8a01f5790 con 0x7ff8a006d8c0 2024-09-17T05:33:23.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.288+0000 7ff88effd640 1 -- 172.21.15.117:0/3477541534 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7ff8980098b0 con 0x7ff8a006d8c0 2024-09-17T05:33:23.289 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:23.289 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":27,"btime":"2024-09-17T05:33:15:876298+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:15.876293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:23.290 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 27 2024-09-17T05:33:23.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.296+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff880078be0 msgr2=0x7ff88007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:23.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.296+0000 7ff89f59e640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff880078be0 0x7ff88007b0f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff89000a830 tx=0x7ff890008040 comp rx=0 tx=0).stop 2024-09-17T05:33:23.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.296+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 msgr2=0x7ff8a0152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:23.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.296+0000 7ff89f59e640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 0x7ff8a0152540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff898009e50 tx=0x7ff898003f30 comp rx=0 tx=0).stop 2024-09-17T05:33:23.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.297+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 shutdown_connections 2024-09-17T05:33:23.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.297+0000 7ff89f59e640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff880078be0 0x7ff88007b0f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:23.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.297+0000 7ff89f59e640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff8a00c5910 0x7ff8a01ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:23.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.297+0000 7ff89f59e640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8a006d8c0 0x7ff8a0152540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:23.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.297+0000 7ff89f59e640 1 --2- 172.21.15.117:0/3477541534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a006cda0 0x7ff8a01f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:23.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.297+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 >> 172.21.15.117:0/3477541534 conn(0x7ff8a00ca0c0 msgr2=0x7ff8a0151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:23.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.301+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 shutdown_connections 2024-09-17T05:33:23.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:23.302+0000 7ff89f59e640 1 -- 172.21.15.117:0/3477541534 wait complete. 2024-09-17T05:33:23.325 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:25.326 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:25.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.593+0000 7f808c6a9640 1 -- 172.21.15.117:0/4008749179 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f808415f530 msgr2=0x7f80841518a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:25.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.593+0000 7f808c6a9640 1 --2- 172.21.15.117:0/4008749179 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f808415f530 0x7f80841518a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f807c005fd0 tx=0x7f807c02a930 comp rx=0 tx=0).stop 2024-09-17T05:33:25.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.597+0000 7f808c6a9640 1 -- 172.21.15.117:0/4008749179 shutdown_connections 2024-09-17T05:33:25.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.597+0000 7f808c6a9640 1 --2- 172.21.15.117:0/4008749179 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f808415f530 0x7f80841518a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:25.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.597+0000 7f808c6a9640 1 --2- 172.21.15.117:0/4008749179 >> v2:172.21.15.119:3300/0 conn(0x7f80840c7810 0x7f808415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:25.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.597+0000 7f808c6a9640 1 -- 172.21.15.117:0/4008749179 >> 172.21.15.117:0/4008749179 conn(0x7f808406d190 msgr2=0x7f808406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:25.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.598+0000 7f808c6a9640 1 -- 172.21.15.117:0/4008749179 shutdown_connections 2024-09-17T05:33:25.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.598+0000 7f808c6a9640 1 -- 172.21.15.117:0/4008749179 wait complete. 2024-09-17T05:33:25.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.601+0000 7f808c6a9640 1 Processor -- start 2024-09-17T05:33:25.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.601+0000 7f808c6a9640 1 -- start start 2024-09-17T05:33:25.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.608+0000 7f808c6a9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80840c7810 0x7f8084148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:25.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.608+0000 7f808c6a9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f808415f120 0x7f808414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.608+0000 7f808c6a9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 0x7f80841fe780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.608+0000 7f808c6a9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f808415e920 con 0x7f808415f120 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.608+0000 7f808c6a9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f80840c8340 con 0x7f808415f530 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.608+0000 7f808c6a9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f8084072e10 con 0x7f80840c7810 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 0x7f80841fe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808a445640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80840c7810 0x7f8084148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:25.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 0x7f80841fe780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44786/0 (socket says 172.21.15.117:44786) 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 -- 172.21.15.117:0/1472553996 learned_addr learned my addr 172.21.15.117:0/1472553996 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f8089c44640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f808415f120 0x7f808414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 -- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80840c7810 msgr2=0x7f8084148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80840c7810 0x7f8084148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 -- 172.21.15.117:0/1472553996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f808415f120 msgr2=0x7f808414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f808415f120 0x7f808414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:25.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f808ac46640 1 -- 172.21.15.117:0/1472553996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8084148dd0 con 0x7f808415f530 2024-09-17T05:33:25.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.610+0000 7f8089c44640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f808415f120 0x7f808414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:33:25.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.611+0000 7f808ac46640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 0x7f80841fe780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f807c006010 tx=0x7f807c030720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:25.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.613+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c030a60 con 0x7f808415f530 2024-09-17T05:33:25.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.613+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f807c03e070 con 0x7f808415f530 2024-09-17T05:33:25.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.613+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c0320d0 con 0x7f808415f530 2024-09-17T05:33:25.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.613+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808414b7d0 con 0x7f808415f530 2024-09-17T05:33:25.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.614+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f808414c1d0 con 0x7f808415f530 2024-09-17T05:33:25.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.617+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f807c032270 con 0x7f808415f530 2024-09-17T05:33:25.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.618+0000 7f807affd640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8068078c60 0x7f806807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:25.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.618+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f807c0bc5b0 con 0x7f808415f530 2024-09-17T05:33:25.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.618+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806c005290 con 0x7f808415f530 2024-09-17T05:33:25.627 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.626+0000 7f808a445640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8068078c60 0x7f806807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:25.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.629+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f807c0846a0 con 0x7f808415f530 2024-09-17T05:33:25.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:25.632+0000 7f808a445640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8068078c60 0x7f806807b170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8074005fd0 tx=0x7f8074005850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:26.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.056+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f806c005580 con 0x7f808415f530 2024-09-17T05:33:26.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.058+0000 7f807affd640 1 -- 172.21.15.117:0/1472553996 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7f807c083fa0 con 0x7f808415f530 2024-09-17T05:33:26.062 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:26.062 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":27,"btime":"2024-09-17T05:33:15:876298+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:15.876293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:26.062 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 27 2024-09-17T05:33:26.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.064+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8068078c60 msgr2=0x7f806807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:26.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.064+0000 7f808c6a9640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8068078c60 0x7f806807b170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8074005fd0 tx=0x7f8074005850 comp rx=0 tx=0).stop 2024-09-17T05:33:26.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.064+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 msgr2=0x7f80841fe780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:26.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.064+0000 7f808c6a9640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 0x7f80841fe780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f807c006010 tx=0x7f807c030720 comp rx=0 tx=0).stop 2024-09-17T05:33:26.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.065+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 shutdown_connections 2024-09-17T05:33:26.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.065+0000 7f808c6a9640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8068078c60 0x7f806807b170 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:26.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.065+0000 7f808c6a9640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f808415f530 0x7f80841fe780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:26.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.065+0000 7f808c6a9640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f808415f120 0x7f808414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:26.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.065+0000 7f808c6a9640 1 --2- 172.21.15.117:0/1472553996 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f80840c7810 0x7f8084148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:26.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.065+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 >> 172.21.15.117:0/1472553996 conn(0x7f808406d190 msgr2=0x7f80840c7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:26.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.070+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 shutdown_connections 2024-09-17T05:33:26.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:26.070+0000 7f808c6a9640 1 -- 172.21.15.117:0/1472553996 wait complete. 2024-09-17T05:33:26.100 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:28.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.322+0000 7fd57ed6d640 1 -- 172.21.15.117:0/2442465767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5640039e0 msgr2=0x7fd564003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.322+0000 7fd57ed6d640 1 --2- 172.21.15.117:0/2442465767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5640039e0 0x7fd564003e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd57806c3f0 tx=0x7fd564005830 comp rx=0 tx=0).stop 2024-09-17T05:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.322+0000 7fd57e56c640 1 -- 172.21.15.117:0/2442465767 reap_dead start 2024-09-17T05:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.323+0000 7fd57ed6d640 1 -- 172.21.15.117:0/2442465767 shutdown_connections 2024-09-17T05:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.323+0000 7fd57ed6d640 1 -- 172.21.15.117:0/2442465767 >> 172.21.15.117:0/2442465767 conn(0x7fd570050ff0 msgr2=0x7fd570051400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.324+0000 7fd57ed6d640 1 -- 172.21.15.117:0/2442465767 shutdown_connections 2024-09-17T05:33:28.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.324+0000 7fd57ed6d640 1 -- 172.21.15.117:0/2442465767 wait complete. 2024-09-17T05:33:28.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.326+0000 7fd57ed6d640 1 Processor -- start 2024-09-17T05:33:28.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.327+0000 7fd57ed6d640 1 -- start start 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.329+0000 7fd57ed6d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 0x7fd5700e09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.329+0000 7fd57dd6b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 0x7fd5700e09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.329+0000 7fd57dd6b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 0x7fd5700e09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50480/0 (socket says 172.21.15.117:50480) 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.329+0000 7fd57dd6b640 1 -- 172.21.15.117:0/3344830324 learned_addr learned my addr 172.21.15.117:0/3344830324 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.329+0000 7fd57ed6d640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd570178de0 0x7fd57004f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.329+0000 7fd57ed6d640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5700d0260 0x7fd5700d0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57ed6d640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700d9300 con 0x7fd5640039e0 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57ed6d640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700dfb00 con 0x7fd5700d0260 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57ed6d640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd570007550 con 0x7fd570178de0 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57e56c640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5700d0260 0x7fd5700d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:28.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57d56a640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd570178de0 0x7fd57004f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57dd6b640 1 -- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd570178de0 msgr2=0x7fd57004f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57dd6b640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd570178de0 0x7fd57004f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57dd6b640 1 -- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5700d0260 msgr2=0x7fd5700d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57dd6b640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5700d0260 0x7fd5700d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57dd6b640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5700d0bb0 con 0x7fd5640039e0 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57d56a640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd570178de0 0x7fd57004f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:33:28.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.330+0000 7fd57dd6b640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 0x7fd5700e09d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd57806c3f0 tx=0x7fd578064270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.331+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5780b4070 con 0x7fd5640039e0 2024-09-17T05:33:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.331+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5780be070 con 0x7fd5640039e0 2024-09-17T05:33:28.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.331+0000 7fd57ed6d640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5700d98e0 con 0x7fd5640039e0 2024-09-17T05:33:28.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.332+0000 7fd57ed6d640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd570047790 con 0x7fd5640039e0 2024-09-17T05:33:28.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.335+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5780af850 con 0x7fd5640039e0 2024-09-17T05:33:28.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.337+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd558005290 con 0x7fd5640039e0 2024-09-17T05:33:28.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.338+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd5780af9f0 con 0x7fd5640039e0 2024-09-17T05:33:28.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.339+0000 7fd56e7fc640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd55c078c60 0x7fd55c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:28.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.339+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fd578137f80 con 0x7fd5640039e0 2024-09-17T05:33:28.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.341+0000 7fd57d56a640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd55c078c60 0x7fd55c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:28.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.342+0000 7fd57d56a640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd55c078c60 0x7fd55c07b170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd570050c60 tx=0x7fd574005da0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:28.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.347+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd5780ff900 con 0x7fd5640039e0 2024-09-17T05:33:28.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.791+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd558005580 con 0x7fd5640039e0 2024-09-17T05:33:28.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.796+0000 7fd56e7fc640 1 -- 172.21.15.117:0/3344830324 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6097 (secure 0 0 0) 0x7fd578104330 con 0x7fd5640039e0 2024-09-17T05:33:28.797 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:28.797 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"btime":"2024-09-17T05:33:27:772180+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:26.818795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:resolve","state_seq":17,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440]},"id":1}]} 2024-09-17T05:33:28.798 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-09-17T05:33:28.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.804+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd55c078c60 msgr2=0x7fd55c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.804+0000 7fd563fff640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd55c078c60 0x7fd55c07b170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd570050c60 tx=0x7fd574005da0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.804+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 msgr2=0x7fd5700e09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.804+0000 7fd563fff640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 0x7fd5700e09d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd57806c3f0 tx=0x7fd578064270 comp rx=0 tx=0).stop 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.805+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 shutdown_connections 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.805+0000 7fd563fff640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd55c078c60 0x7fd55c07b170 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.805+0000 7fd563fff640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5700d0260 0x7fd5700d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.805+0000 7fd563fff640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd570178de0 0x7fd57004f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.805+0000 7fd563fff640 1 --2- 172.21.15.117:0/3344830324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5640039e0 0x7fd5700e09d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:28.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.805+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 >> 172.21.15.117:0/3344830324 conn(0x7fd570050ff0 msgr2=0x7fd5700d2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:28.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.808+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 shutdown_connections 2024-09-17T05:33:28.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:28.808+0000 7fd563fff640 1 -- 172.21.15.117:0/3344830324 wait complete. 2024-09-17T05:33:28.831 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:29.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:33:29.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:33:29.350 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-09-17T05:33:29.349+0000 7fc10f28f640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52318) UID: 0 2024-09-17T05:33:29.352 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-09-17T05:33:29.350+0000 7f06b4d87640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52318) UID: 0 2024-09-17T05:33:29.353 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-09-17T05:33:29.352+0000 7f163881d640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52318) UID: 0 2024-09-17T05:33:29.359 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-09-17T05:33:29.350+0000 7fe00adf6640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 52318) UID: 0 2024-09-17T05:33:30.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:31.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.042+0000 7f973b442640 1 -- 172.21.15.117:0/3021194228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c045b70 msgr2=0x7f972c0d8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:31.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.042+0000 7f973b442640 1 --2- 172.21.15.117:0/3021194228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c045b70 0x7f972c0d8b80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f972c0df040 tx=0x7f972402a5c0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.042+0000 7f973b442640 1 -- 172.21.15.117:0/3021194228 shutdown_connections 2024-09-17T05:33:31.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.042+0000 7f973b442640 1 --2- 172.21.15.117:0/3021194228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c045b70 0x7f972c0d8b80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.042+0000 7f973b442640 1 -- 172.21.15.117:0/3021194228 >> 172.21.15.117:0/3021194228 conn(0x7f972c0506d0 msgr2=0x7f972c050ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:31.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.043+0000 7f973b442640 1 -- 172.21.15.117:0/3021194228 shutdown_connections 2024-09-17T05:33:31.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.043+0000 7f973b442640 1 -- 172.21.15.117:0/3021194228 wait complete. 2024-09-17T05:33:31.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.046+0000 7f973b442640 1 Processor -- start 2024-09-17T05:33:31.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.046+0000 7f973b442640 1 -- start start 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973b442640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972c046d90 0x7f972c0d5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973b442640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c0d90c0 0x7f972c0dbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973b442640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f972c005240 0x7f972c005820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973b442640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f972c0471a0 con 0x7f972c0d90c0 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973b442640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f972c0e2f40 con 0x7f972c046d90 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973b442640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f972c035f80 con 0x7f972c005240 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973a440640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972c046d90 0x7f972c0d5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f9739c3f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c0d90c0 0x7f972c0dbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:31.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973ac41640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f972c005240 0x7f972c005820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:31.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973ac41640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f972c005240 0x7f972c005820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:49624/0 (socket says 172.21.15.117:49624) 2024-09-17T05:33:31.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.049+0000 7f973ac41640 1 -- 172.21.15.117:0/2183716206 learned_addr learned my addr 172.21.15.117:0/2183716206 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:31.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.050+0000 7f9739c3f640 1 -- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f972c005240 msgr2=0x7f972c005820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:31.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.050+0000 7f9739c3f640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f972c005240 0x7f972c005820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.050+0000 7f9739c3f640 1 -- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972c046d90 msgr2=0x7f972c0d5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:31.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.050+0000 7f9739c3f640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972c046d90 0x7f972c0d5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.050+0000 7f9739c3f640 1 -- 172.21.15.117:0/2183716206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c0d5770 con 0x7f972c0d90c0 2024-09-17T05:33:31.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.050+0000 7f9739c3f640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c0d90c0 0x7f972c0dbdd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f972c0e28b0 tx=0x7f9724032f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:31.052 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.051+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724041030 con 0x7f972c0d90c0 2024-09-17T05:33:31.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.051+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972c17a130 con 0x7f972c0d90c0 2024-09-17T05:33:31.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.051+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f972c0d6f80 con 0x7f972c0d90c0 2024-09-17T05:33:31.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.052+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9724005020 con 0x7f972c0d90c0 2024-09-17T05:33:31.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.052+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240053f0 con 0x7f972c0d90c0 2024-09-17T05:33:31.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.054+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971c005290 con 0x7f972c0d90c0 2024-09-17T05:33:31.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.057+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9724039040 con 0x7f972c0d90c0 2024-09-17T05:33:31.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.058+0000 7f972affd640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9718078c60 0x7f971807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:31.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.058+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f97240bc460 con 0x7f972c0d90c0 2024-09-17T05:33:31.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.063+0000 7f973a440640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9718078c60 0x7f971807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:31.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.066+0000 7f973a440640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9718078c60 0x7f971807b170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f973406b840 tx=0x7f9734070040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:31.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.067+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f972404b030 con 0x7f972c0d90c0 2024-09-17T05:33:31.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.537+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f971c005580 con 0x7f972c0d90c0 2024-09-17T05:33:31.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.539+0000 7f972affd640 1 -- 172.21.15.117:0/2183716206 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6097 (secure 0 0 0) 0x7f97240844c0 con 0x7f972c0d90c0 2024-09-17T05:33:31.540 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:31.540 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"btime":"2024-09-17T05:33:27:772180+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:26.818795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:resolve","state_seq":17,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440]},"id":1}]} 2024-09-17T05:33:31.540 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.542+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9718078c60 msgr2=0x7f971807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9718078c60 0x7f971807b170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f973406b840 tx=0x7f9734070040 comp rx=0 tx=0).stop 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c0d90c0 msgr2=0x7f972c0dbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c0d90c0 0x7f972c0dbdd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f972c0e28b0 tx=0x7f9724032f00 comp rx=0 tx=0).stop 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 shutdown_connections 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9718078c60 0x7f971807b170 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f972c005240 0x7f972c005820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f972c0d90c0 0x7f972c0dbdd0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 --2- 172.21.15.117:0/2183716206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972c046d90 0x7f972c0d5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:31.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.543+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 >> 172.21.15.117:0/2183716206 conn(0x7f972c0506d0 msgr2=0x7f972c04c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:31.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.544+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 shutdown_connections 2024-09-17T05:33:31.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:31.544+0000 7f973b442640 1 -- 172.21.15.117:0/2183716206 wait complete. 2024-09-17T05:33:31.566 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:33:27:772180+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 46 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:33:26.818795+0000", "qdb_cluster": [ 5082, 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:33:33.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:33.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.787+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2549143148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d50004d10 msgr2=0x7f1d50005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:33.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.787+0000 7f1d6e09b640 1 --2- 172.21.15.117:0/2549143148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d50004d10 0x7f1d50005160 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c006f10 tx=0x7f1d50005830 comp rx=0 tx=0).stop 2024-09-17T05:33:33.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.790+0000 7f1d67fff640 1 -- 172.21.15.117:0/2549143148 reap_dead start 2024-09-17T05:33:33.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.792+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2549143148 shutdown_connections 2024-09-17T05:33:33.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.792+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2549143148 >> 172.21.15.117:0/2549143148 conn(0x7f1d68074d80 msgr2=0x7f1d680d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:33.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.792+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2549143148 shutdown_connections 2024-09-17T05:33:33.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.792+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2549143148 wait complete. 2024-09-17T05:33:33.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.795+0000 7f1d6e09b640 1 Processor -- start 2024-09-17T05:33:33.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.797+0000 7f1d6e09b640 1 -- start start 2024-09-17T05:33:33.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.799+0000 7f1d6e09b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 0x7f1d681580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:33.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.799+0000 7f1d677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 0x7f1d681580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.799+0000 7f1d677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 0x7f1d681580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50524/0 (socket says 172.21.15.117:50524) 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.799+0000 7f1d677fe640 1 -- 172.21.15.117:0/2907999371 learned_addr learned my addr 172.21.15.117:0/2907999371 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.800+0000 7f1d6e09b640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d68159bd0 0x7f1d68156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.800+0000 7f1d6e09b640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1d68159fe0 0x7f1d68157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.800+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6815c180 con 0x7f1d68158c90 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.800+0000 7f1d66ffd640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d68159bd0 0x7f1d68156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.800+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6806e4d0 con 0x7f1d68159bd0 2024-09-17T05:33:33.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.800+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f1d680cd0d0 con 0x7f1d68159fe0 2024-09-17T05:33:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.801+0000 7f1d677fe640 1 -- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1d68159fe0 msgr2=0x7f1d68157160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:33:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.801+0000 7f1d677fe640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1d68159fe0 0x7f1d68157160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.801+0000 7f1d677fe640 1 -- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d68159bd0 msgr2=0x7f1d68156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.801+0000 7f1d677fe640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d68159bd0 0x7f1d68156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.801+0000 7f1d677fe640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d680c6ee0 con 0x7f1d68158c90 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.801+0000 7f1d677fe640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 0x7f1d681580b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00c290 tx=0x7f1d5c006650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.802+0000 7f1d66ffd640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d68159bd0 0x7f1d68156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.802+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c032030 con 0x7f1d68158c90 2024-09-17T05:33:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.803+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d5c004b10 con 0x7f1d68158c90 2024-09-17T05:33:33.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.803+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d68155d10 con 0x7f1d68158c90 2024-09-17T05:33:33.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.805+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c039440 con 0x7f1d68158c90 2024-09-17T05:33:33.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.808+0000 7f1d6e09b640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d681562d0 con 0x7f1d68158c90 2024-09-17T05:33:33.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.815+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1d5c035020 con 0x7f1d68158c90 2024-09-17T05:33:33.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.815+0000 7f1d4ffff640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1d54078c60 0x7f1d5407b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:33.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.816+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f1d5c0bae30 con 0x7f1d68158c90 2024-09-17T05:33:33.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.817+0000 7f1d66ffd640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1d54078c60 0x7f1d5407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:33.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.818+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d680cb330 con 0x7f1d68158c90 2024-09-17T05:33:33.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.832+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1d5c082e80 con 0x7f1d68158c90 2024-09-17T05:33:33.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:33.848+0000 7f1d66ffd640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1d54078c60 0x7f1d5407b170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1d680b6330 tx=0x7f1d58006a20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:34.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.293+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d680b37b0 con 0x7f1d68158c90 2024-09-17T05:33:34.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.294+0000 7f1d4ffff640 1 -- 172.21.15.117:0/2907999371 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6099 (secure 0 0 0) 0x7f1d5c082780 con 0x7f1d68158c90 2024-09-17T05:33:34.296 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:34.296 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":29,"btime":"2024-09-17T05:33:32:815194+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:32.789012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:34.296 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 29 2024-09-17T05:33:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1d54078c60 msgr2=0x7f1d5407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1d54078c60 0x7f1d5407b170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1d680b6330 tx=0x7f1d58006a20 comp rx=0 tx=0).stop 2024-09-17T05:33:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 msgr2=0x7f1d681580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:34.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 0x7f1d681580b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00c290 tx=0x7f1d5c006650 comp rx=0 tx=0).stop 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 shutdown_connections 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1d54078c60 0x7f1d5407b170 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1d68159fe0 0x7f1d68157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d68159bd0 0x7f1d68156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 --2- 172.21.15.117:0/2907999371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d68158c90 0x7f1d681580b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.301+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 >> 172.21.15.117:0/2907999371 conn(0x7f1d68074d80 msgr2=0x7f1d68159260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.302+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 shutdown_connections 2024-09-17T05:33:34.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:34.302+0000 7f1d4dffb640 1 -- 172.21.15.117:0/2907999371 wait complete. 2024-09-17T05:33:34.331 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:36.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:36.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.551+0000 7f391f313640 1 -- 172.21.15.117:0/2887966877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3904003c50 msgr2=0x7f39040040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:36.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.551+0000 7f391f313640 1 --2- 172.21.15.117:0/2887966877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3904003c50 0x7f39040040e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f39180746b0 tx=0x7f3904005830 comp rx=0 tx=0).stop 2024-09-17T05:33:36.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.552+0000 7f391d8b0640 1 -- 172.21.15.117:0/2887966877 reap_dead start 2024-09-17T05:33:36.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.555+0000 7f391f313640 1 -- 172.21.15.117:0/2887966877 shutdown_connections 2024-09-17T05:33:36.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.555+0000 7f391f313640 1 -- 172.21.15.117:0/2887966877 >> 172.21.15.117:0/2887966877 conn(0x7f391806f0e0 msgr2=0x7f3918070720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:36.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.557+0000 7f391f313640 1 -- 172.21.15.117:0/2887966877 shutdown_connections 2024-09-17T05:33:36.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.558+0000 7f391f313640 1 -- 172.21.15.117:0/2887966877 wait complete. 2024-09-17T05:33:36.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.561+0000 7f391f313640 1 Processor -- start 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.561+0000 7f391f313640 1 -- start start 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.564+0000 7f391f313640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3904003c50 0x7f39180b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.564+0000 7f391f313640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 0x7f39181ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.564+0000 7f391f313640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3918155aa0 0x7f39181e4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.564+0000 7f391c8ae640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 0x7f39181ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391c8ae640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 0x7f39181ef290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57508/0 (socket says 172.21.15.117:57508) 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391d8b0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3918155aa0 0x7f39181e4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391d8b0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3918155aa0 0x7f39181e4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52932/0 (socket says 172.21.15.117:52932) 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391d0af640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3904003c50 0x7f39180b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391d0af640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3904003c50 0x7f39180b90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:57564/0 (socket says 172.21.15.117:57564) 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391c8ae640 1 -- 172.21.15.117:0/537062176 learned_addr learned my addr 172.21.15.117:0/537062176 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:36.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391f313640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39180c5b50 con 0x7f3918155aa0 2024-09-17T05:33:36.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391f313640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f39180c4120 con 0x7f3918158340 2024-09-17T05:33:36.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.565+0000 7f391f313640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f39180c7a60 con 0x7f3904003c50 2024-09-17T05:33:36.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.568+0000 7f391c8ae640 1 -- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3904003c50 msgr2=0x7f39180b90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:36.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.568+0000 7f391c8ae640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3904003c50 0x7f39180b90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:36.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.568+0000 7f391c8ae640 1 -- 172.21.15.117:0/537062176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3918155aa0 msgr2=0x7f39181e4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:36.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.568+0000 7f391c8ae640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3918155aa0 0x7f39181e4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:36.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.568+0000 7f391c8ae640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39181ee810 con 0x7f3918158340 2024-09-17T05:33:36.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.570+0000 7f391c8ae640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 0x7f39181ef290 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3908003d60 tx=0x7f3908001800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:36.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.570+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908011640 con 0x7f3918158340 2024-09-17T05:33:36.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.571+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3908016020 con 0x7f3918158340 2024-09-17T05:33:36.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.571+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3908011640 con 0x7f3918158340 2024-09-17T05:33:36.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.571+0000 7f391f313640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39180cca70 con 0x7f3918158340 2024-09-17T05:33:36.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.571+0000 7f391f313640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39181eea90 con 0x7f3918158340 2024-09-17T05:33:36.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.573+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f4005290 con 0x7f3918158340 2024-09-17T05:33:36.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.576+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3908011800 con 0x7f3918158340 2024-09-17T05:33:36.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.577+0000 7f390dffb640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38fc07b000 0x7f38fc07d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:36.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.578+0000 7f391d0af640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38fc07b000 0x7f38fc07d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:36.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.578+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f3908099d80 con 0x7f3918158340 2024-09-17T05:33:36.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.584+0000 7f391d0af640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38fc07b000 0x7f38fc07d510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3914035000 tx=0x7f391402d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:36.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:36.585+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3908020020 con 0x7f3918158340 2024-09-17T05:33:37.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.020+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38f4005580 con 0x7f3918158340 2024-09-17T05:33:37.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.021+0000 7f390dffb640 1 -- 172.21.15.117:0/537062176 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6099 (secure 0 0 0) 0x7f39181ee810 con 0x7f3918158340 2024-09-17T05:33:37.030 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:37.030 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":29,"btime":"2024-09-17T05:33:32:815194+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:32.789012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082]},"id":1}]} 2024-09-17T05:33:37.030 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 29 2024-09-17T05:33:37.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.032+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38fc07b000 msgr2=0x7f38fc07d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:37.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.032+0000 7f38fb7fe640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38fc07b000 0x7f38fc07d510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3914035000 tx=0x7f391402d040 comp rx=0 tx=0).stop 2024-09-17T05:33:37.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.032+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 msgr2=0x7f39181ef290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:37.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 0x7f39181ef290 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3908003d60 tx=0x7f3908001800 comp rx=0 tx=0).stop 2024-09-17T05:33:37.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 shutdown_connections 2024-09-17T05:33:37.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f38fc07b000 0x7f38fc07d510 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3914035000 tx=0x7f391402d040 comp rx=0 tx=0).stop 2024-09-17T05:33:37.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3918155aa0 0x7f39181e4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:37.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3918158340 0x7f39181ef290 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:37.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 --2- 172.21.15.117:0/537062176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3904003c50 0x7f39180b90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:37.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.033+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 >> 172.21.15.117:0/537062176 conn(0x7f391806f0e0 msgr2=0x7f3918153e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:37.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.037+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 shutdown_connections 2024-09-17T05:33:37.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:37.037+0000 7f38fb7fe640 1 -- 172.21.15.117:0/537062176 wait complete. 2024-09-17T05:33:37.058 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:39.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.264+0000 7f2a88980640 1 -- 172.21.15.117:0/774797567 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0047a0 con 0x7f2a840c4040 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/774797567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840c4040 msgr2=0x7f2a841557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/774797567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840c4040 0x7f2a841557f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2a840c8e60 tx=0x7f2a7c02a5c0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/774797567 shutdown_connections 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/774797567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840c4040 0x7f2a841557f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/774797567 >> 172.21.15.117:0/774797567 conn(0x7f2a84076430 msgr2=0x7f2a84072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:39.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/774797567 shutdown_connections 2024-09-17T05:33:39.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.265+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/774797567 wait complete. 2024-09-17T05:33:39.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.268+0000 7f2a8bbe6640 1 Processor -- start 2024-09-17T05:33:39.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.268+0000 7f2a8bbe6640 1 -- start start 2024-09-17T05:33:39.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.271+0000 7f2a8bbe6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a840c4040 0x7f2a840cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:39.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.271+0000 7f2a8bbe6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 0x7f2a840c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.271+0000 7f2a89181640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 0x7f2a840c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.271+0000 7f2a89181640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 0x7f2a840c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52948/0 (socket says 172.21.15.117:52948) 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.272+0000 7f2a89181640 1 -- 172.21.15.117:0/555609031 learned_addr learned my addr 172.21.15.117:0/555609031 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.272+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a840cb640 0x7f2a840c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.272+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8406e8c0 con 0x7f2a840cb230 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.272+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2a840cba50 con 0x7f2a840cb640 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.272+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2a840c6820 con 0x7f2a840c4040 2024-09-17T05:33:39.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.272+0000 7f2a8a183640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a840cb640 0x7f2a840c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.273+0000 7f2a89181640 1 -- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a840c4040 msgr2=0x7f2a840cd030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.273+0000 7f2a89181640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a840c4040 0x7f2a840cd030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.273+0000 7f2a89181640 1 -- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a840cb640 msgr2=0x7f2a840c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.273+0000 7f2a89181640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a840cb640 0x7f2a840c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.273+0000 7f2a89181640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a840c7ac0 con 0x7f2a840cb230 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.273+0000 7f2a89181640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 0x7f2a840c5ef0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000bd90 tx=0x7f2a80009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.274+0000 7f2a8a183640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a840cb640 0x7f2a840c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.274+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80019070 con 0x7f2a840cb230 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.274+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a8000cd90 con 0x7f2a840cb230 2024-09-17T05:33:39.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.274+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a840c7da0 con 0x7f2a840cb230 2024-09-17T05:33:39.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.277+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a840c8360 con 0x7f2a840cb230 2024-09-17T05:33:39.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.277+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a800146a0 con 0x7f2a840cb230 2024-09-17T05:33:39.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.278+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a8001c070 con 0x7f2a840cb230 2024-09-17T05:33:39.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.278+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a841557f0 con 0x7f2a840cb230 2024-09-17T05:33:39.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.281+0000 7f2a7a7fc640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a68078c60 0x7f2a6807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:39.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.281+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f2a8009aea0 con 0x7f2a840cb230 2024-09-17T05:33:39.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.281+0000 7f2a89982640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a68078c60 0x7f2a6807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:39.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.282+0000 7f2a89982640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a68078c60 0x7f2a6807b170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c033000 tx=0x7f2a7c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:39.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.287+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a80062f60 con 0x7f2a840cb230 2024-09-17T05:33:39.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.712+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a840caa90 con 0x7f2a840cb230 2024-09-17T05:33:39.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.713+0000 7f2a7a7fc640 1 -- 172.21.15.117:0/555609031 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6096 (secure 0 0 0) 0x7f2a80062860 con 0x7f2a840cb230 2024-09-17T05:33:39.715 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:39.715 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":30,"btime":"2024-09-17T05:33:37:853720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:37.835239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440]},"id":1}]} 2024-09-17T05:33:39.715 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 30 2024-09-17T05:33:39.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a68078c60 msgr2=0x7f2a6807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a68078c60 0x7f2a6807b170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c033000 tx=0x7f2a7c004730 comp rx=0 tx=0).stop 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 msgr2=0x7f2a840c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 0x7f2a840c5ef0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000bd90 tx=0x7f2a80009f10 comp rx=0 tx=0).stop 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 shutdown_connections 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2a68078c60 0x7f2a6807b170 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a840cb640 0x7f2a840c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a840cb230 0x7f2a840c5ef0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 --2- 172.21.15.117:0/555609031 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2a840c4040 0x7f2a840cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:39.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.716+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 >> 172.21.15.117:0/555609031 conn(0x7f2a84076430 msgr2=0x7f2a84159110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:39.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.717+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 shutdown_connections 2024-09-17T05:33:39.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:39.717+0000 7f2a8bbe6640 1 -- 172.21.15.117:0/555609031 wait complete. 2024-09-17T05:33:39.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:41.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:41.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.950+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/363434107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b28163c00 msgr2=0x7f2b28166020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:41.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.951+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/363434107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b28163c00 0x7f2b28166020 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2b20005fd0 tx=0x7f2b2002aaa0 comp rx=0 tx=0).stop 2024-09-17T05:33:41.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.952+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/363434107 shutdown_connections 2024-09-17T05:33:41.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.952+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/363434107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b28163c00 0x7f2b28166020 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:41.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.952+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/363434107 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b280c92c0 0x7f2b280c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:41.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.952+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/363434107 >> 172.21.15.117:0/363434107 conn(0x7f2b2806d190 msgr2=0x7f2b2806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:41.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.955+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/363434107 shutdown_connections 2024-09-17T05:33:41.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.956+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/363434107 wait complete. 2024-09-17T05:33:41.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.960+0000 7f2b2d7a8640 1 Processor -- start 2024-09-17T05:33:41.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.960+0000 7f2b2d7a8640 1 -- start start 2024-09-17T05:33:41.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.962+0000 7f2b2d7a8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b280c92c0 0x7f2b280ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:41.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.962+0000 7f2b2d7a8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 0x7f2b280cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:41.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.962+0000 7f2b2d7a8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b28163c00 0x7f2b280cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b2d7a8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2815c120 con 0x7f2b2815bd10 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b2d7a8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2b28152090 con 0x7f2b28163c00 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b2d7a8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2b28072e10 con 0x7f2b280c92c0 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b1ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 0x7f2b280cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b28163c00 0x7f2b280cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b277fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b28163c00 0x7f2b280cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57526/0 (socket says 172.21.15.117:57526) 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b1ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 0x7f2b280cb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52966/0 (socket says 172.21.15.117:52966) 2024-09-17T05:33:41.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.963+0000 7f2b277fe640 1 -- 172.21.15.117:0/2110476594 learned_addr learned my addr 172.21.15.117:0/2110476594 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:41.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.964+0000 7f2b26ffd640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b280c92c0 0x7f2b280ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.966+0000 7f2b1ffff640 1 -- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b280c92c0 msgr2=0x7f2b280ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.966+0000 7f2b1ffff640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b280c92c0 0x7f2b280ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.966+0000 7f2b1ffff640 1 -- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b28163c00 msgr2=0x7f2b280cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.966+0000 7f2b1ffff640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b28163c00 0x7f2b280cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.966+0000 7f2b1ffff640 1 -- 172.21.15.117:0/2110476594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b280cacf0 con 0x7f2b2815bd10 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.966+0000 7f2b1ffff640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 0x7f2b280cb6a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2b280bd8c0 tx=0x7f2b1800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.967+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18015020 con 0x7f2b2815bd10 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.967+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2815a5d0 con 0x7f2b2815bd10 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.967+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b280c4ca0 con 0x7f2b2815bd10 2024-09-17T05:33:41.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.967+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b180119e0 con 0x7f2b2815bd10 2024-09-17T05:33:41.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.970+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18011e30 con 0x7f2b2815bd10 2024-09-17T05:33:41.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.971+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b280c6850 con 0x7f2b2815bd10 2024-09-17T05:33:41.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.971+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2b18018020 con 0x7f2b2815bd10 2024-09-17T05:33:41.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.972+0000 7f2b1f7fe640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b08078c60 0x7f2b0807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:41.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.972+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f2b1809aff0 con 0x7f2b2815bd10 2024-09-17T05:33:41.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.973+0000 7f2b26ffd640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b08078c60 0x7f2b0807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:41.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.979+0000 7f2b26ffd640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b08078c60 0x7f2b0807b170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2b10006fd0 tx=0x7f2b10009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:41.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:41.981+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2b1801f020 con 0x7f2b2815bd10 2024-09-17T05:33:42.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.392+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b280c49f0 con 0x7f2b2815bd10 2024-09-17T05:33:42.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.396+0000 7f2b1f7fe640 1 -- 172.21.15.117:0/2110476594 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6096 (secure 0 0 0) 0x7f2b180631f0 con 0x7f2b2815bd10 2024-09-17T05:33:42.401 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:42.401 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":30,"btime":"2024-09-17T05:33:37:853720+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:37.835239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5082,4440]},"id":1}]} 2024-09-17T05:33:42.401 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 30 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.403+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b08078c60 msgr2=0x7f2b0807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.403+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b08078c60 0x7f2b0807b170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2b10006fd0 tx=0x7f2b10009060 comp rx=0 tx=0).stop 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.403+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 msgr2=0x7f2b280cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.403+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 0x7f2b280cb6a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2b280bd8c0 tx=0x7f2b1800b870 comp rx=0 tx=0).stop 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 shutdown_connections 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b08078c60 0x7f2b0807b170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b28163c00 0x7f2b280cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2815bd10 0x7f2b280cb6a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:42.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 --2- 172.21.15.117:0/2110476594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b280c92c0 0x7f2b280ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:42.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 >> 172.21.15.117:0/2110476594 conn(0x7f2b2806d190 msgr2=0x7f2b2814ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:42.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 shutdown_connections 2024-09-17T05:33:42.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:42.404+0000 7f2b2d7a8640 1 -- 172.21.15.117:0/2110476594 wait complete. 2024-09-17T05:33:42.428 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:33:44.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.643+0000 7fb6df429640 1 -- 172.21.15.117:0/1068876233 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d8150b30 msgr2=0x7fb6d8150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.643+0000 7fb6df429640 1 --2- 172.21.15.117:0/1068876233 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d8150b30 0x7fb6d8150f40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0005fd0 tx=0x7fb6d002a820 comp rx=0 tx=0).stop 2024-09-17T05:33:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.644+0000 7fb6df429640 1 -- 172.21.15.117:0/1068876233 shutdown_connections 2024-09-17T05:33:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.644+0000 7fb6df429640 1 --2- 172.21.15.117:0/1068876233 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d8150b30 0x7fb6d8150f40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.644+0000 7fb6df429640 1 --2- 172.21.15.117:0/1068876233 >> v2:172.21.15.119:3300/0 conn(0x7fb6d806f500 0x7fb6d80ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.644+0000 7fb6df429640 1 -- 172.21.15.117:0/1068876233 >> 172.21.15.117:0/1068876233 conn(0x7fb6d80713f0 msgr2=0x7fb6d8072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.645+0000 7fb6df429640 1 -- 172.21.15.117:0/1068876233 shutdown_connections 2024-09-17T05:33:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.646+0000 7fb6df429640 1 -- 172.21.15.117:0/1068876233 wait complete. 2024-09-17T05:33:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.649+0000 7fb6df429640 1 Processor -- start 2024-09-17T05:33:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.649+0000 7fb6df429640 1 -- start start 2024-09-17T05:33:44.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6df429640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 0x7fb6d80c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:44.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6df429640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d806f500 0x7fb6d814ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:44.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6df429640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6d8150b30 0x7fb6d8145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:44.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6df429640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d80c9980 con 0x7fb6d806f500 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6de427640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 0x7fb6d80c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6de427640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 0x7fb6d80c5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:35964/0 (socket says 172.21.15.117:35964) 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6de427640 1 -- 172.21.15.117:0/286347948 learned_addr learned my addr 172.21.15.117:0/286347948 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.652+0000 7fb6dec28640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6d8150b30 0x7fb6d8145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.653+0000 7fb6ddc26640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d806f500 0x7fb6d814ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.653+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d80c4350 con 0x7fb6d8150b30 2024-09-17T05:33:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.653+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d80cecd0 con 0x7fb6d806e5b0 2024-09-17T05:33:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.654+0000 7fb6de427640 1 -- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6d8150b30 msgr2=0x7fb6d8145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.654+0000 7fb6de427640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6d8150b30 0x7fb6d8145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.654+0000 7fb6de427640 1 -- 172.21.15.117:0/286347948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d806f500 msgr2=0x7fb6d814ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.654+0000 7fb6de427640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d806f500 0x7fb6d814ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.654+0000 7fb6de427640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d80c6120 con 0x7fb6d806e5b0 2024-09-17T05:33:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.656+0000 7fb6de427640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 0x7fb6d80c5be0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c3f0 tx=0x7fb6d400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.657+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4017070 con 0x7fb6d806e5b0 2024-09-17T05:33:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.657+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6d40035f0 con 0x7fb6d806e5b0 2024-09-17T05:33:44.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.657+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4003a40 con 0x7fb6d806e5b0 2024-09-17T05:33:44.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.657+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d815d910 con 0x7fb6d806e5b0 2024-09-17T05:33:44.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.657+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d815eb70 con 0x7fb6d806e5b0 2024-09-17T05:33:44.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.661+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb6d401a070 con 0x7fb6d806e5b0 2024-09-17T05:33:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.663+0000 7fb6ceffd640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb6bc078c60 0x7fb6bc07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:44.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.659+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c0005290 con 0x7fb6d806e5b0 2024-09-17T05:33:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.665+0000 7fb6ddc26640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb6bc078c60 0x7fb6bc07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.666+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fb6d409a2c0 con 0x7fb6d806e5b0 2024-09-17T05:33:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.666+0000 7fb6ddc26640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb6bc078c60 0x7fb6bc07b170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8154e70 tx=0x7fb6c8009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:44.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:44.675+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb6d4061d60 con 0x7fb6d806e5b0 2024-09-17T05:33:45.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.143+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6c0005580 con 0x7fb6d806e5b0 2024-09-17T05:33:45.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.144+0000 7fb6ceffd640 1 -- 172.21.15.117:0/286347948 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6101 (secure 0 0 0) 0x7fb6d4066790 con 0x7fb6d806e5b0 2024-09-17T05:33:45.148 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:45.148 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":31,"btime":"2024-09-17T05:33:44:899428+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:44.899419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082,5232]},"id":1}]} 2024-09-17T05:33:45.148 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 31 2024-09-17T05:33:45.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.150+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb6bc078c60 msgr2=0x7fb6bc07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.150+0000 7fb6df429640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb6bc078c60 0x7fb6bc07b170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8154e70 tx=0x7fb6c8009060 comp rx=0 tx=0).stop 2024-09-17T05:33:45.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.150+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 msgr2=0x7fb6d80c5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.150+0000 7fb6df429640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 0x7fb6d80c5be0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c3f0 tx=0x7fb6d400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.151+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 shutdown_connections 2024-09-17T05:33:45.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.151+0000 7fb6df429640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb6bc078c60 0x7fb6bc07b170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.151+0000 7fb6df429640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6d8150b30 0x7fb6d8145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.151+0000 7fb6df429640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d806f500 0x7fb6d814ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.151+0000 7fb6df429640 1 --2- 172.21.15.117:0/286347948 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6d806e5b0 0x7fb6d80c5be0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.151+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 >> 172.21.15.117:0/286347948 conn(0x7fb6d80713f0 msgr2=0x7fb6d80cc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:45.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.154+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 shutdown_connections 2024-09-17T05:33:45.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.154+0000 7fb6df429640 1 -- 172.21.15.117:0/286347948 wait complete. 2024-09-17T05:33:45.178 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=4434 2024-09-17T05:33:45.178 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2024-09-17T05:33:45.178 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-09-17T05:33:45.179 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T05:33:45.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T05:33:45.181 INFO:tasks.ceph.mds.b:Started 2024-09-17T05:33:45.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:45.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.390+0000 7f09827c3640 1 -- 172.21.15.117:0/1929495148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 msgr2=0x7f097c14d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.390+0000 7f09827c3640 1 --2- 172.21.15.117:0/1929495148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c14d1f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0974005fd0 tx=0x7f097402aaa0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.391+0000 7f09827c3640 1 -- 172.21.15.117:0/1929495148 shutdown_connections 2024-09-17T05:33:45.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.391+0000 7f09827c3640 1 --2- 172.21.15.117:0/1929495148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c14d1f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.391+0000 7f09827c3640 1 --2- 172.21.15.117:0/1929495148 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c0c6f10 0x7f097c15b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.391+0000 7f09827c3640 1 -- 172.21.15.117:0/1929495148 >> 172.21.15.117:0/1929495148 conn(0x7f097c06fbe0 msgr2=0x7f097c0706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:45.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.399+0000 7f09827c3640 1 -- 172.21.15.117:0/1929495148 shutdown_connections 2024-09-17T05:33:45.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.399+0000 7f09827c3640 1 -- 172.21.15.117:0/1929495148 wait complete. 2024-09-17T05:33:45.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.402+0000 7f09827c3640 1 Processor -- start 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.402+0000 7f09827c3640 1 -- start start 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.404+0000 7f09827c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c074240 0x7f097c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.404+0000 7f09827c3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f097c0c6f10 0x7f097c0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.405+0000 7f097bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c074240 0x7f097c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.405+0000 7f09827c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.405+0000 7f09827c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f097c0c7ec0 con 0x7f097c15b5b0 2024-09-17T05:33:45.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.405+0000 7f09827c3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f097c0c7be0 con 0x7f097c0c6f10 2024-09-17T05:33:45.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.405+0000 7f09827c3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f097c0c81a0 con 0x7f097c074240 2024-09-17T05:33:45.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.405+0000 7f097b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f097c0c6f10 0x7f097c0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:45.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.406+0000 7f097bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c074240 0x7f097c0cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:35994/0 (socket says 172.21.15.117:35994) 2024-09-17T05:33:45.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.406+0000 7f097bfff640 1 -- 172.21.15.117:0/585129632 learned_addr learned my addr 172.21.15.117:0/585129632 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:45.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.406+0000 7f0980d60640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.408+0000 7f0980d60640 1 -- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c074240 msgr2=0x7f097c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.408+0000 7f0980d60640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c074240 0x7f097c0cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.408+0000 7f0980d60640 1 -- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f097c0c6f10 msgr2=0x7f097c0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.408+0000 7f0980d60640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f097c0c6f10 0x7f097c0c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.408+0000 7f0980d60640 1 -- 172.21.15.117:0/585129632 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c1fe0f0 con 0x7f097c15b5b0 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.408+0000 7f097b7fe640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f097c0c6f10 0x7f097c0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.409+0000 7f0980d60640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c15ea30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f097c0c36d0 tx=0x7f09740045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.410+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974036400 con 0x7f097c15b5b0 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.410+0000 7f09827c3640 1 -- 172.21.15.117:0/585129632 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c157630 con 0x7f097c15b5b0 2024-09-17T05:33:45.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.410+0000 7f09827c3640 1 -- 172.21.15.117:0/585129632 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c0cbed0 con 0x7f097c15b5b0 2024-09-17T05:33:45.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.411+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09740365a0 con 0x7f097c15b5b0 2024-09-17T05:33:45.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.413+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09740369b0 con 0x7f097c15b5b0 2024-09-17T05:33:45.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.414+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c005290 con 0x7f097c15b5b0 2024-09-17T05:33:45.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.424+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f097402c040 con 0x7f097c15b5b0 2024-09-17T05:33:45.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.425+0000 7f0978ff9640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0968078c60 0x7f096807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:45.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.425+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f09740bccd0 con 0x7f097c15b5b0 2024-09-17T05:33:45.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.426+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f09740c2020 con 0x7f097c15b5b0 2024-09-17T05:33:45.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.426+0000 7f097bfff640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0968078c60 0x7f096807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:45.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.433+0000 7f097bfff640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0968078c60 0x7f096807b170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0970006fd0 tx=0x7f0970008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:45.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.874+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f095c005580 con 0x7f097c15b5b0 2024-09-17T05:33:45.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.880+0000 7f0978ff9640 1 -- 172.21.15.117:0/585129632 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6101 (secure 0 0 0) 0x7f0974032030 con 0x7f097c15b5b0 2024-09-17T05:33:45.883 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:45.883 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":31,"btime":"2024-09-17T05:33:44:899428+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:44.899419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082,5232]},"id":1}]} 2024-09-17T05:33:45.883 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 31 2024-09-17T05:33:45.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.885+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0968078c60 msgr2=0x7f096807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.886+0000 7f09627fc640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0968078c60 0x7f096807b170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0970006fd0 tx=0x7f0970008040 comp rx=0 tx=0).stop 2024-09-17T05:33:45.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.886+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 msgr2=0x7f097c15ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:45.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.886+0000 7f09627fc640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c15ea30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f097c0c36d0 tx=0x7f09740045e0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 shutdown_connections 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0968078c60 0x7f096807b170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f097c15b5b0 0x7f097c15ea30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f097c0c6f10 0x7f097c0c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 --2- 172.21.15.117:0/585129632 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f097c074240 0x7f097c0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 >> 172.21.15.117:0/585129632 conn(0x7f097c06fbe0 msgr2=0x7f097c148540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 shutdown_connections 2024-09-17T05:33:45.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:45.890+0000 7f09627fc640 1 -- 172.21.15.117:0/585129632 wait complete. 2024-09-17T05:33:45.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-17T05:33:46.668 INFO:tasks.ceph.mds.f.smithi119.stderr:2024-09-17T05:33:46.667+0000 7f6f66926640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-17T05:33:47.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:48.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.139+0000 7f08ab80e640 1 -- 172.21.15.117:0/3765645732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a40c6f50 msgr2=0x7f08a4073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.139+0000 7f08ab80e640 1 --2- 172.21.15.117:0/3765645732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a40c6f50 0x7f08a4073f60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f08a4073500 tx=0x7f089402a4d0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.140+0000 7f08ab80e640 1 -- 172.21.15.117:0/3765645732 shutdown_connections 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.140+0000 7f08ab80e640 1 --2- 172.21.15.117:0/3765645732 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a415b130 0x7f08a40b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.140+0000 7f08ab80e640 1 --2- 172.21.15.117:0/3765645732 >> v2:172.21.15.119:3300/0 conn(0x7f08a40c8390 0x7f08a415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.140+0000 7f08ab80e640 1 --2- 172.21.15.117:0/3765645732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a40c6f50 0x7f08a4073f60 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f08a4073500 tx=0x7f089402a4d0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.140+0000 7f08ab80e640 1 -- 172.21.15.117:0/3765645732 >> 172.21.15.117:0/3765645732 conn(0x7f08a40c9a20 msgr2=0x7f08a40c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:48.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.141+0000 7f08ab80e640 1 -- 172.21.15.117:0/3765645732 shutdown_connections 2024-09-17T05:33:48.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.142+0000 7f08ab80e640 1 -- 172.21.15.117:0/3765645732 wait complete. 2024-09-17T05:33:48.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.145+0000 7f08ab80e640 1 Processor -- start 2024-09-17T05:33:48.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.145+0000 7f08ab80e640 1 -- start start 2024-09-17T05:33:48.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.147+0000 7f08ab80e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 0x7f08a414a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:48.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.147+0000 7f08ab80e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08a415b130 0x7f08a4148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:48.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.147+0000 7f08ab80e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a414ada0 0x7f08a41547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:48.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.147+0000 7f08ab80e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4157dd0 con 0x7f08a414ada0 2024-09-17T05:33:48.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.147+0000 7f08ab80e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4157af0 con 0x7f08a415b130 2024-09-17T05:33:48.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.147+0000 7f08ab80e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f08a40c87a0 con 0x7f08a40c8390 2024-09-17T05:33:48.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.148+0000 7f08a95aa640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 0x7f08a414a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:48.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.148+0000 7f08a95aa640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 0x7f08a414a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:36010/0 (socket says 172.21.15.117:36010) 2024-09-17T05:33:48.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.148+0000 7f08a95aa640 1 -- 172.21.15.117:0/1203216609 learned_addr learned my addr 172.21.15.117:0/1203216609 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:48.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.148+0000 7f08a8da9640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08a415b130 0x7f08a4148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:48.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.149+0000 7f08a9dab640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a414ada0 0x7f08a41547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:48.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.150+0000 7f08a95aa640 1 -- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08a415b130 msgr2=0x7f08a4148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:48.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.150+0000 7f08a95aa640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08a415b130 0x7f08a4148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.150+0000 7f08a95aa640 1 -- 172.21.15.117:0/1203216609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a414ada0 msgr2=0x7f08a41547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:48.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.150+0000 7f08a95aa640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a414ada0 0x7f08a41547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.150+0000 7f08a95aa640 1 -- 172.21.15.117:0/1203216609 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a4149500 con 0x7f08a40c8390 2024-09-17T05:33:48.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.152+0000 7f08a95aa640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 0x7f08a414a5b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0894009950 tx=0x7f0894032b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:48.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.155+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894040070 con 0x7f08a40c8390 2024-09-17T05:33:48.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.155+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0894005310 con 0x7f08a40c8390 2024-09-17T05:33:48.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.155+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08940047a0 con 0x7f08a40c8390 2024-09-17T05:33:48.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.156+0000 7f08ab80e640 1 -- 172.21.15.117:0/1203216609 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a41595a0 con 0x7f08a40c8390 2024-09-17T05:33:48.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.156+0000 7f08ab80e640 1 -- 172.21.15.117:0/1203216609 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a415a690 con 0x7f08a40c8390 2024-09-17T05:33:48.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.160+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0894039040 con 0x7f08a40c8390 2024-09-17T05:33:48.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.160+0000 7f0899ffb640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0888078c60 0x7f088807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:48.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.160+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f08940bc180 con 0x7f08a40c8390 2024-09-17T05:33:48.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.162+0000 7f08a8da9640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0888078c60 0x7f088807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:48.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.162+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a40c7e70 con 0x7f08a40c8390 2024-09-17T05:33:48.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.165+0000 7f08a8da9640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0888078c60 0x7f088807b170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08a00096c0 tx=0x7f08a0011040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:48.179 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.174+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f08a40c7e70 con 0x7f08a40c8390 2024-09-17T05:33:48.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.636+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08a40c94d0 con 0x7f08a40c8390 2024-09-17T05:33:48.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.641+0000 7f0899ffb640 1 -- 172.21.15.117:0/1203216609 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6101 (secure 0 0 0) 0x7f0894083c20 con 0x7f08a40c8390 2024-09-17T05:33:48.645 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:48.645 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":31,"btime":"2024-09-17T05:33:44:899428+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:44.899419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5082,5232]},"id":1}]} 2024-09-17T05:33:48.645 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 31 2024-09-17T05:33:48.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0888078c60 msgr2=0x7f088807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:48.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0888078c60 0x7f088807b170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08a00096c0 tx=0x7f08a0011040 comp rx=0 tx=0).stop 2024-09-17T05:33:48.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 msgr2=0x7f08a414a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:48.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 0x7f08a414a5b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0894009950 tx=0x7f0894032b70 comp rx=0 tx=0).stop 2024-09-17T05:33:48.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 shutdown_connections 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0888078c60 0x7f088807b170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08a414ada0 0x7f08a41547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08a415b130 0x7f08a4148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 --2- 172.21.15.117:0/1203216609 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08a40c8390 0x7f08a414a5b0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0894009950 tx=0x7f0894032b70 comp rx=0 tx=0).stop 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.648+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 >> 172.21.15.117:0/1203216609 conn(0x7f08a40c9a20 msgr2=0x7f08a414baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.649+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 shutdown_connections 2024-09-17T05:33:48.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:48.649+0000 7f08877fe640 1 -- 172.21.15.117:0/1203216609 wait complete. 2024-09-17T05:33:48.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-17T05:33:50.620 INFO:tasks.ceph.mds.b.smithi119.stdout:starting mds.b at 2024-09-17T05:33:50.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:33:50.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.879+0000 7f815bfff640 1 -- 172.21.15.117:0/1229628642 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815000c260 con 0x7f8144004800 2024-09-17T05:33:50.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.880+0000 7f8162700640 1 -- 172.21.15.117:0/1229628642 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 msgr2=0x7f8144004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:50.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.880+0000 7f8162700640 1 --2- 172.21.15.117:0/1229628642 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 0x7f8144004c10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f815000c610 tx=0x7f8144005830 comp rx=0 tx=0).stop 2024-09-17T05:33:50.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.880+0000 7f8162700640 1 -- 172.21.15.117:0/1229628642 shutdown_connections 2024-09-17T05:33:50.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.880+0000 7f8162700640 1 --2- 172.21.15.117:0/1229628642 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 0x7f8144004c10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:50.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.880+0000 7f8162700640 1 -- 172.21.15.117:0/1229628642 >> 172.21.15.117:0/1229628642 conn(0x7f815c0c8d10 msgr2=0x7f815c0c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:50.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.881+0000 7f8162700640 1 -- 172.21.15.117:0/1229628642 shutdown_connections 2024-09-17T05:33:50.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.881+0000 7f8162700640 1 -- 172.21.15.117:0/1229628642 wait complete. 2024-09-17T05:33:50.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.884+0000 7f8162700640 1 Processor -- start 2024-09-17T05:33:50.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.884+0000 7f8162700640 1 -- start start 2024-09-17T05:33:50.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.886+0000 7f8162700640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 0x7f815c0c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.886+0000 7f8162700640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f815c0becd0 0x7f815c14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.886+0000 7f8162700640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f815c150110 0x7f815c1f9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.886+0000 7f8162700640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f815c073160 con 0x7f815c150110 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.886+0000 7f8162700640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f815c15bfa0 con 0x7f815c0becd0 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.886+0000 7f8162700640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f815c073fc0 con 0x7f8144004800 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.887+0000 7f81616fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 0x7f815c0c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.887+0000 7f8160efd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f815c0becd0 0x7f815c14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.887+0000 7f8161eff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f815c150110 0x7f815c1f9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.887+0000 7f8161eff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f815c150110 0x7f815c1f9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44614/0 (socket says 172.21.15.117:44614) 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.887+0000 7f8161eff640 1 -- 172.21.15.117:0/1471067612 learned_addr learned my addr 172.21.15.117:0/1471067612 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.888+0000 7f8160efd640 1 -- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 msgr2=0x7f815c0c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.888+0000 7f8160efd640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 0x7f815c0c1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.888+0000 7f8160efd640 1 -- 172.21.15.117:0/1471067612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f815c150110 msgr2=0x7f815c1f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.888+0000 7f8160efd640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f815c150110 0x7f815c1f9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.888+0000 7f8160efd640 1 -- 172.21.15.117:0/1471067612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815c0c2350 con 0x7f815c0becd0 2024-09-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.888+0000 7f8160efd640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f815c0becd0 0x7f815c14fbd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8150030000 tx=0x7f8150006290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.889+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8150004050 con 0x7f815c0becd0 2024-09-17T05:33:50.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.889+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81500041f0 con 0x7f815c0becd0 2024-09-17T05:33:50.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.889+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8150004050 con 0x7f815c0becd0 2024-09-17T05:33:50.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.889+0000 7f8162700640 1 -- 172.21.15.117:0/1471067612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815c15c550 con 0x7f815c0becd0 2024-09-17T05:33:50.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.891+0000 7f8162700640 1 -- 172.21.15.117:0/1471067612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f815c0c1330 con 0x7f815c0becd0 2024-09-17T05:33:50.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.893+0000 7f8162700640 1 -- 172.21.15.117:0/1471067612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815c0bdbc0 con 0x7f815c0becd0 2024-09-17T05:33:50.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.895+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f815002b670 con 0x7f815c0becd0 2024-09-17T05:33:50.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.896+0000 7f8159ffb640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8148078fc0 0x7f814807b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:33:50.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.896+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f815003b030 con 0x7f815c0becd0 2024-09-17T05:33:50.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.896+0000 7f81616fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8148078fc0 0x7f814807b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:33:50.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.900+0000 7f81616fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8148078fc0 0x7f814807b4d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8154003fd0 tx=0x7f8154006ed0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:33:50.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:50.906+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f815c0bdbc0 con 0x7f815c0becd0 2024-09-17T05:33:51.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.352+0000 7f8162700640 1 -- 172.21.15.117:0/1471067612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f815c06eeb0 con 0x7f815c0becd0 2024-09-17T05:33:51.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.353+0000 7f8159ffb640 1 -- 172.21.15.117:0/1471067612 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6929 (secure 0 0 0) 0x7f815c06eeb0 con 0x7f815c0becd0 2024-09-17T05:33:51.358 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:33:51.358 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":32,"btime":"2024-09-17T05:33:50:930815+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:44.899419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5232,5082,4440]},"id":1}]} 2024-09-17T05:33:51.358 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 32 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.361+0000 7f81437fe640 1 -- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8148078fc0 msgr2=0x7f814807b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.361+0000 7f81437fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8148078fc0 0x7f814807b4d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8154003fd0 tx=0x7f8154006ed0 comp rx=0 tx=0).stop 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.361+0000 7f81437fe640 1 -- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f815c0becd0 msgr2=0x7f815c14fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.361+0000 7f81437fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f815c0becd0 0x7f815c14fbd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8150030000 tx=0x7f8150006290 comp rx=0 tx=0).stop 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.362+0000 7f81437fe640 1 -- 172.21.15.117:0/1471067612 shutdown_connections 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.362+0000 7f81437fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8148078fc0 0x7f814807b4d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.362+0000 7f81437fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f815c150110 0x7f815c1f9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.362+0000 7f81437fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f815c0becd0 0x7f815c14fbd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:51.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.362+0000 7f81437fe640 1 --2- 172.21.15.117:0/1471067612 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8144004800 0x7f815c0c1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:33:51.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.362+0000 7f81437fe640 1 -- 172.21.15.117:0/1471067612 >> 172.21.15.117:0/1471067612 conn(0x7f815c0c8d10 msgr2=0x7f815c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:33:51.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.365+0000 7f81437fe640 1 -- 172.21.15.117:0/1471067612 shutdown_connections 2024-09-17T05:33:51.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:33:51.366+0000 7f81437fe640 1 -- 172.21.15.117:0/1471067612 wait complete. 2024-09-17T05:33:51.387 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-09-17T05:33:51.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2024-09-17T05:34:00.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:00.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.598+0000 7f602b59e640 1 -- 172.21.15.117:0/3832870219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c0c7ff0 msgr2=0x7f602c0c74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:00.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.598+0000 7f602b59e640 1 --2- 172.21.15.117:0/3832870219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c0c7ff0 0x7f602c0c74f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6024005fd0 tx=0x7f602402ccf0 comp rx=0 tx=0).stop 2024-09-17T05:34:00.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.598+0000 7f602b59e640 1 -- 172.21.15.117:0/3832870219 shutdown_connections 2024-09-17T05:34:00.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.598+0000 7f602b59e640 1 --2- 172.21.15.117:0/3832870219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c0c7ff0 0x7f602c0c74f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:00.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.598+0000 7f602b59e640 1 --2- 172.21.15.117:0/3832870219 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c161ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:00.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.598+0000 7f602b59e640 1 -- 172.21.15.117:0/3832870219 >> 172.21.15.117:0/3832870219 conn(0x7f602c0c8d10 msgr2=0x7f602c0c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.599+0000 7f602b59e640 1 -- 172.21.15.117:0/3832870219 shutdown_connections 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.599+0000 7f602b59e640 1 -- 172.21.15.117:0/3832870219 wait complete. 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.602+0000 7f602b59e640 1 Processor -- start 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.602+0000 7f602b59e640 1 -- start start 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f602b59e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c08c830 0x7f602c156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f602b59e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f602b59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f602c16ad20 0x7f602c15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f602b59e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f602c16bca0 con 0x7f602c16ad20 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f602b59e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0c6cf0 con 0x7f602c08c830 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f602b59e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f602c16be20 con 0x7f602c0c4080 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f6029d9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:00.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f6029d9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c15ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:58824/0 (socket says 172.21.15.117:58824) 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.604+0000 7f6029d9b640 1 -- 172.21.15.117:0/3204961721 learned_addr learned my addr 172.21.15.117:0/3204961721 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f602ad9d640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f602c16ad20 0x7f602c15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f6029d9b640 1 -- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c08c830 msgr2=0x7f602c156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f6029d9b640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c08c830 0x7f602c156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f6029d9b640 1 -- 172.21.15.117:0/3204961721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f602c16ad20 msgr2=0x7f602c15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f6029d9b640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f602c16ad20 0x7f602c15ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f6029d9b640 1 -- 172.21.15.117:0/3204961721 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602c08b090 con 0x7f602c0c4080 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.605+0000 7f602ad9d640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f602c16ad20 0x7f602c15ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.606+0000 7f6029d9b640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c15ae30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6024009bf0 tx=0x7f602400e4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:00.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.609+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602403a040 con 0x7f602c0c4080 2024-09-17T05:34:00.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.609+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602c163520 con 0x7f602c0c4080 2024-09-17T05:34:00.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.609+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f602c157910 con 0x7f602c0c4080 2024-09-17T05:34:00.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.611+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60240359a0 con 0x7f602c0c4080 2024-09-17T05:34:00.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.611+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024035db0 con 0x7f602c0c4080 2024-09-17T05:34:00.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.613+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6010005290 con 0x7f602c0c4080 2024-09-17T05:34:00.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.615+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f602403e420 con 0x7f602c0c4080 2024-09-17T05:34:00.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.616+0000 7f601affd640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6014078da0 0x7f601407b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:00.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.616+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f60240bbbe0 con 0x7f602c0c4080 2024-09-17T05:34:00.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.616+0000 7f602a59c640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6014078da0 0x7f601407b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:00.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.618+0000 7f602a59c640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6014078da0 0x7f601407b2b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f601c00da60 tx=0x7f601c00d520 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:00.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:00.624+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f602404c020 con 0x7f602c0c4080 2024-09-17T05:34:01.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.091+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6010005580 con 0x7f602c0c4080 2024-09-17T05:34:01.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.093+0000 7f601affd640 1 -- 172.21.15.117:0/3204961721 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6929 (secure 0 0 0) 0x7f6024083cd0 con 0x7f602c0c4080 2024-09-17T05:34:01.096 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:01.096 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":32,"btime":"2024-09-17T05:33:50:930815+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:33:44.899419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[5232,5082,4440]},"id":1}]} 2024-09-17T05:34:01.096 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 32 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.099+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6014078da0 msgr2=0x7f601407b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.099+0000 7f602b59e640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6014078da0 0x7f601407b2b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f601c00da60 tx=0x7f601c00d520 comp rx=0 tx=0).stop 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.099+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 msgr2=0x7f602c15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c15ae30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6024009bf0 tx=0x7f602400e4c0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 shutdown_connections 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6014078da0 0x7f601407b2b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f602c16ad20 0x7f602c15ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f602c0c4080 0x7f602c15ae30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 --2- 172.21.15.117:0/3204961721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f602c08c830 0x7f602c156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 >> 172.21.15.117:0/3204961721 conn(0x7f602c0c8d10 msgr2=0x7f602c0738b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:01.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 shutdown_connections 2024-09-17T05:34:01.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.100+0000 7f602b59e640 1 -- 172.21.15.117:0/3204961721 wait complete. 2024-09-17T05:34:01.121 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 1 2024-09-17T05:34:01.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-09-17T05:34:01.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.285+0000 7f4c5780a640 1 -- 172.21.15.117:0/4041759116 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c500c9b60 msgr2=0x7f4c50159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.285+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4041759116 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c500c9b60 0x7f4c50159a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0097e0 tx=0x7f4c4c02a4d0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.286+0000 7f4c5780a640 1 -- 172.21.15.117:0/4041759116 shutdown_connections 2024-09-17T05:34:01.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.286+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4041759116 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c500c9b60 0x7f4c50159a00 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0097e0 tx=0x7f4c4c02a4d0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.287+0000 7f4c5780a640 1 -- 172.21.15.117:0/4041759116 >> 172.21.15.117:0/4041759116 conn(0x7f4c500ccbe0 msgr2=0x7f4c500c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:01.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.288+0000 7f4c5780a640 1 -- 172.21.15.117:0/4041759116 shutdown_connections 2024-09-17T05:34:01.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.291+0000 7f4c5780a640 1 -- 172.21.15.117:0/4041759116 wait complete. 2024-09-17T05:34:01.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.294+0000 7f4c5780a640 1 Processor -- start 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.294+0000 7f4c5780a640 1 -- start start 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.296+0000 7f4c5780a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c5006cf80 0x7f4c500c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.296+0000 7f4c5780a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 0x7f4c501e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c54da5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 0x7f4c501e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c54da5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 0x7f4c501e12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37818/0 (socket says 172.21.15.117:37818) 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c54da5640 1 -- 172.21.15.117:0/4278851288 learned_addr learned my addr 172.21.15.117:0/4278851288 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c500c56e0 0x7f4c501e1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4c500c73b0 con 0x7f4c5006d390 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c555a6640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c5006cf80 0x7f4c500c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.297+0000 7f4c55da7640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c500c56e0 0x7f4c501e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c500753d0 con 0x7f4c500c56e0 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4c50153590 con 0x7f4c5006cf80 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c54da5640 1 -- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c5006cf80 msgr2=0x7f4c500c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c54da5640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c5006cf80 0x7f4c500c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c54da5640 1 -- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c500c56e0 msgr2=0x7f4c501e1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c54da5640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c500c56e0 0x7f4c501e1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.298+0000 7f4c54da5640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c500c4830 con 0x7f4c5006d390 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.299+0000 7f4c54da5640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 0x7f4c501e12d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c780 tx=0x7f4c4000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.299+0000 7f4c55da7640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c500c56e0 0x7f4c501e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.299+0000 7f4c555a6640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c5006cf80 0x7f4c500c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.299+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40005270 con 0x7f4c5006d390 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.300+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c40005410 con 0x7f4c5006d390 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.300+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000a8a0 con 0x7f4c5006d390 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.300+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c501e4180 con 0x7f4c5006d390 2024-09-17T05:34:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.300+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c500c3af0 con 0x7f4c5006d390 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.303+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c500c2ae0 con 0x7f4c5006d390 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.304+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4c4000aa40 con 0x7f4c5006d390 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.305+0000 7f4c45ffb640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4c34078be0 0x7f4c3407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.305+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f4c4009ad90 con 0x7f4c5006d390 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.308+0000 7f4c555a6640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4c34078be0 0x7f4c3407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.314+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4c500c2ae0 con 0x7f4c5006d390 2024-09-17T05:34:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.315+0000 7f4c555a6640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4c34078be0 0x7f4c3407b0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c034000 tx=0x7f4c4c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:01.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.775+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7f4c50159a00 con 0x7f4c5006d390 2024-09-17T05:34:01.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.975+0000 7f4c45ffb640 1 -- 172.21.15.117:0/4278851288 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v33) ==== 105+0+0 (secure 0 0 0) 0x7f4c40062df0 con 0x7f4c5006d390 2024-09-17T05:34:01.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4c34078be0 msgr2=0x7f4c3407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4c34078be0 0x7f4c3407b0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c034000 tx=0x7f4c4c004730 comp rx=0 tx=0).stop 2024-09-17T05:34:01.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 msgr2=0x7f4c501e12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:01.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 0x7f4c501e12d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c780 tx=0x7f4c4000cc40 comp rx=0 tx=0).stop 2024-09-17T05:34:01.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 shutdown_connections 2024-09-17T05:34:01.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4c34078be0 0x7f4c3407b0f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c500c56e0 0x7f4c501e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c5006d390 0x7f4c501e12d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 --2- 172.21.15.117:0/4278851288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4c5006cf80 0x7f4c500c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:01.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.980+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 >> 172.21.15.117:0/4278851288 conn(0x7f4c500ccbe0 msgr2=0x7f4c501e4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:01.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.983+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 shutdown_connections 2024-09-17T05:34:01.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:01.983+0000 7f4c5780a640 1 -- 172.21.15.117:0/4278851288 wait complete. 2024-09-17T05:34:02.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-09-17T05:34:02.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 -- 172.21.15.117:0/4268271202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c15bd70 msgr2=0x7fe57c0ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 --2- 172.21.15.117:0/4268271202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c15bd70 0x7fe57c0ca770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe564005fd0 tx=0x7fe56402a5c0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 -- 172.21.15.117:0/4268271202 shutdown_connections 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 --2- 172.21.15.117:0/4268271202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe57c0b52b0 0x7fe57c0b56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 --2- 172.21.15.117:0/4268271202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c15bd70 0x7fe57c0ca770 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 -- 172.21.15.117:0/4268271202 >> 172.21.15.117:0/4268271202 conn(0x7fe57c074d80 msgr2=0x7fe57c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:02.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 -- 172.21.15.117:0/4268271202 shutdown_connections 2024-09-17T05:34:02.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.242+0000 7fe580a0e640 1 -- 172.21.15.117:0/4268271202 wait complete. 2024-09-17T05:34:02.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.246+0000 7fe580a0e640 1 Processor -- start 2024-09-17T05:34:02.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.246+0000 7fe580a0e640 1 -- start start 2024-09-17T05:34:02.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe580a0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 0x7fe57c15d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe580a0e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe57c0b52b0 0x7fe57c1f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe580a0e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe57c15bd70 0x7fe57c152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe580a0e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c0cacb0 con 0x7fe57c06dbe0 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe580a0e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c0cd0d0 con 0x7fe57c0b52b0 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe580a0e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c1558b0 con 0x7fe57c15bd70 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.250+0000 7fe57a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 0x7fe57c15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:02.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 0x7fe57c15d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37836/0 (socket says 172.21.15.117:37836) 2024-09-17T05:34:02.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 -- 172.21.15.117:0/1055130032 learned_addr learned my addr 172.21.15.117:0/1055130032 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:02.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 -- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe57c15bd70 msgr2=0x7fe57c152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:02.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe57c15bd70 0x7fe57c152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 -- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe57c0b52b0 msgr2=0x7fe57c1f4dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:34:02.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe57c0b52b0 0x7fe57c1f4dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.251+0000 7fe57a59c640 1 -- 172.21.15.117:0/1055130032 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c1f5310 con 0x7fe57c06dbe0 2024-09-17T05:34:02.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.252+0000 7fe57a59c640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 0x7fe57c15d570 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe564009850 tx=0x7fe564003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:02.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.258+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56403e030 con 0x7fe57c06dbe0 2024-09-17T05:34:02.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.258+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe564003e00 con 0x7fe57c06dbe0 2024-09-17T05:34:02.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.258+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564035040 con 0x7fe57c06dbe0 2024-09-17T05:34:02.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.259+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c155c80 con 0x7fe57c06dbe0 2024-09-17T05:34:02.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.259+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c156200 con 0x7fe57c06dbe0 2024-09-17T05:34:02.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.262+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe57c0cb290 con 0x7fe57c06dbe0 2024-09-17T05:34:02.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.263+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe564041030 con 0x7fe57c06dbe0 2024-09-17T05:34:02.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.264+0000 7fe572ffd640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe55c078c90 0x7fe55c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:02.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.264+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fe5640bb940 con 0x7fe57c06dbe0 2024-09-17T05:34:02.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.264+0000 7fe579d9b640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe55c078c90 0x7fe55c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:02.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.268+0000 7fe579d9b640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe55c078c90 0x7fe55c07b1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe56c005c40 tx=0x7fe56c005bd0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:02.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.272+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe564083910 con 0x7fe57c06dbe0 2024-09-17T05:34:02.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.727+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe57c06ee70 con 0x7fe57c06dbe0 2024-09-17T05:34:02.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.730+0000 7fe572ffd640 1 -- 172.21.15.117:0/1055130032 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7fe564083210 con 0x7fe57c06dbe0 2024-09-17T05:34:02.734 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:02.734 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:02.734 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:02.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.737+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe55c078c90 msgr2=0x7fe55c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:02.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.737+0000 7fe580a0e640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe55c078c90 0x7fe55c07b1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe56c005c40 tx=0x7fe56c005bd0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.737+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 msgr2=0x7fe57c15d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:02.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.737+0000 7fe580a0e640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 0x7fe57c15d570 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe564009850 tx=0x7fe564003c10 comp rx=0 tx=0).stop 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 shutdown_connections 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe55c078c90 0x7fe55c07b1a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe57c15bd70 0x7fe57c152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe57c0b52b0 0x7fe57c1f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 --2- 172.21.15.117:0/1055130032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe57c06dbe0 0x7fe57c15d570 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 >> 172.21.15.117:0/1055130032 conn(0x7fe57c074d80 msgr2=0x7fe57c154bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.738+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 shutdown_connections 2024-09-17T05:34:02.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:02.739+0000 7fe580a0e640 1 -- 172.21.15.117:0/1055130032 wait complete. 2024-09-17T05:34:02.760 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-09-17T05:34:02.761 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:34:01:978270+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 46 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:34:01.978252+0000", "qdb_cluster": [ 4440, 5232 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:34:04.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.974+0000 7f39824cc640 1 -- 172.21.15.117:0/1968449192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3964005150 msgr2=0x7f396400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.974+0000 7f39824cc640 1 --2- 172.21.15.117:0/1968449192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3964005150 0x7f396400b9c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f397c0753a0 tx=0x7f3964005830 comp rx=0 tx=0).stop 2024-09-17T05:34:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.974+0000 7f39824cc640 1 -- 172.21.15.117:0/1968449192 shutdown_connections 2024-09-17T05:34:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.974+0000 7f39824cc640 1 --2- 172.21.15.117:0/1968449192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3964005150 0x7f396400b9c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.974+0000 7f39824cc640 1 -- 172.21.15.117:0/1968449192 >> 172.21.15.117:0/1968449192 conn(0x7f397c06d190 msgr2=0x7f397c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:04.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.976+0000 7f39824cc640 1 -- 172.21.15.117:0/1968449192 shutdown_connections 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.979+0000 7f39824cc640 1 -- 172.21.15.117:0/1968449192 wait complete. 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.982+0000 7f39824cc640 1 Processor -- start 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.982+0000 7f39824cc640 1 -- start start 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.984+0000 7f39824cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3964005150 0x7f397c0c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.984+0000 7f39824cc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f397c158a30 0x7f397c0cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.984+0000 7f39824cc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 0x7f397c0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f39824cc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f397c15c120 con 0x7f397c0cb560 2024-09-17T05:34:04.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f39824cc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f397c152090 con 0x7f3964005150 2024-09-17T05:34:04.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f39824cc640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f397c072e10 con 0x7f397c158a30 2024-09-17T05:34:04.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f3980a69640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 0x7f397c0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f3980a69640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 0x7f397c0cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56510/0 (socket says 172.21.15.117:56510) 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f3980a69640 1 -- 172.21.15.117:0/3460955505 learned_addr learned my addr 172.21.15.117:0/3460955505 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.985+0000 7f397b7fe640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f397c158a30 0x7f397c0cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.986+0000 7f3980a69640 1 -- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f397c158a30 msgr2=0x7f397c0cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.986+0000 7f3980a69640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f397c158a30 0x7f397c0cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.986+0000 7f3980a69640 1 -- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3964005150 msgr2=0x7f397c0c38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.986+0000 7f3980a69640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3964005150 0x7f397c0c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.986+0000 7f3980a69640 1 -- 172.21.15.117:0/3460955505 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c0c3e10 con 0x7f397c0cb560 2024-09-17T05:34:04.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.986+0000 7f3980a69640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 0x7f397c0cb970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f396c00c350 tx=0x7f396c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:04.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.987+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c032040 con 0x7f397c0cb560 2024-09-17T05:34:04.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.987+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f396c004d30 con 0x7f397c0cb560 2024-09-17T05:34:04.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.987+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c02e6c0 con 0x7f397c0cb560 2024-09-17T05:34:04.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.987+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397c143f80 con 0x7f397c0cb560 2024-09-17T05:34:04.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.987+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397c15e5c0 con 0x7f397c0cb560 2024-09-17T05:34:04.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.991+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f396c035020 con 0x7f397c0cb560 2024-09-17T05:34:04.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.992+0000 7f3978ff9640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3968078c60 0x7f396807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:04.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.993+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f396c0bb880 con 0x7f397c0cb560 2024-09-17T05:34:04.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.994+0000 7f397bfff640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3968078c60 0x7f396807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:04.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:04.998+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395c005290 con 0x7f397c0cb560 2024-09-17T05:34:05.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.000+0000 7f397bfff640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3968078c60 0x7f396807b170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3974005c40 tx=0x7f3974005bd0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:05.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.009+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f396c0838e0 con 0x7f397c0cb560 2024-09-17T05:34:05.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.452+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f395c005580 con 0x7f397c0cb560 2024-09-17T05:34:05.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.453+0000 7f3978ff9640 1 -- 172.21.15.117:0/3460955505 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f396c02de80 con 0x7f397c0cb560 2024-09-17T05:34:05.457 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:05.457 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:05.458 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:05.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.457+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3968078c60 msgr2=0x7f396807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:05.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.457+0000 7f39824cc640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3968078c60 0x7f396807b170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3974005c40 tx=0x7f3974005bd0 comp rx=0 tx=0).stop 2024-09-17T05:34:05.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.457+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 msgr2=0x7f397c0cb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:05.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.457+0000 7f39824cc640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 0x7f397c0cb970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f396c00c350 tx=0x7f396c004600 comp rx=0 tx=0).stop 2024-09-17T05:34:05.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.458+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 shutdown_connections 2024-09-17T05:34:05.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.458+0000 7f39824cc640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3968078c60 0x7f396807b170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:05.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.458+0000 7f39824cc640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f397c0cb560 0x7f397c0cb970 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:05.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.458+0000 7f39824cc640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f397c158a30 0x7f397c0cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:05.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.458+0000 7f39824cc640 1 --2- 172.21.15.117:0/3460955505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3964005150 0x7f397c0c38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:05.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.458+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 >> 172.21.15.117:0/3460955505 conn(0x7f397c06d190 msgr2=0x7f397c159c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:05.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.460+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 shutdown_connections 2024-09-17T05:34:05.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:05.460+0000 7f39824cc640 1 -- 172.21.15.117:0/3460955505 wait complete. 2024-09-17T05:34:05.479 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:07.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:07.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.669+0000 7fef36358640 1 -- 172.21.15.117:0/2806270736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef28007190 msgr2=0x7fef28034300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:07.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.669+0000 7fef36358640 1 --2- 172.21.15.117:0/2806270736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef28007190 0x7fef28034300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fef2000b9c0 tx=0x7fef2002c210 comp rx=0 tx=0).stop 2024-09-17T05:34:07.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.672+0000 7fef36358640 1 -- 172.21.15.117:0/2806270736 shutdown_connections 2024-09-17T05:34:07.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.672+0000 7fef36358640 1 --2- 172.21.15.117:0/2806270736 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef280ffda0 0x7fef281001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:07.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.672+0000 7fef36358640 1 --2- 172.21.15.117:0/2806270736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef28007190 0x7fef28034300 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:07.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.672+0000 7fef36358640 1 -- 172.21.15.117:0/2806270736 >> 172.21.15.117:0/2806270736 conn(0x7fef2804fe30 msgr2=0x7fef28050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:07.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.672+0000 7fef36358640 1 -- 172.21.15.117:0/2806270736 shutdown_connections 2024-09-17T05:34:07.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.673+0000 7fef36358640 1 -- 172.21.15.117:0/2806270736 wait complete. 2024-09-17T05:34:07.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.675+0000 7fef36358640 1 Processor -- start 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.676+0000 7fef36358640 1 -- start start 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.678+0000 7fef36358640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef28007190 0x7fef280fecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.678+0000 7fef36358640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef2804c180 0x7fef2819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.678+0000 7fef36358640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef280ffda0 0x7fef2819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.678+0000 7fef36358640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef2804bba0 con 0x7fef2804c180 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.678+0000 7fef36358640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef28006990 con 0x7fef280ffda0 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.678+0000 7fef36358640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fef2804bd20 con 0x7fef28007190 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.679+0000 7fef35b57640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef280ffda0 0x7fef2819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.679+0000 7fef35b57640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef280ffda0 0x7fef2819fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34384/0 (socket says 172.21.15.117:34384) 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.679+0000 7fef35b57640 1 -- 172.21.15.117:0/184081892 learned_addr learned my addr 172.21.15.117:0/184081892 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.679+0000 7fef35356640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef28007190 0x7fef280fecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:07.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.680+0000 7fef35356640 1 -- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef280ffda0 msgr2=0x7fef2819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:07.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.680+0000 7fef35356640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef280ffda0 0x7fef2819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:07.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.680+0000 7fef35356640 1 -- 172.21.15.117:0/184081892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef2804c180 msgr2=0x7fef2819f6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:34:07.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.680+0000 7fef35356640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef2804c180 0x7fef2819f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:07.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.680+0000 7fef35356640 1 -- 172.21.15.117:0/184081892 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef281a0160 con 0x7fef28007190 2024-09-17T05:34:07.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.680+0000 7fef35356640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef28007190 0x7fef280fecd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00c450 tx=0x7fef2c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:07.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.681+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c017070 con 0x7fef28007190 2024-09-17T05:34:07.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.681+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef280f3ac0 con 0x7fef28007190 2024-09-17T05:34:07.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.682+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef28185bd0 con 0x7fef28007190 2024-09-17T05:34:07.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.682+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef2c005480 con 0x7fef28007190 2024-09-17T05:34:07.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.683+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c017070 con 0x7fef28007190 2024-09-17T05:34:07.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.684+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef10005290 con 0x7fef28007190 2024-09-17T05:34:07.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.689+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef2c01a070 con 0x7fef28007190 2024-09-17T05:34:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.689+0000 7fef25ffb640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef14078f90 0x7fef1407b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.690+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fef2c005280 con 0x7fef28007190 2024-09-17T05:34:07.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.694+0000 7fef34b55640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef14078f90 0x7fef1407b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:07.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.695+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef2c066820 con 0x7fef28007190 2024-09-17T05:34:07.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:07.696+0000 7fef34b55640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef14078f90 0x7fef1407b4a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef20036000 tx=0x7fef20004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:08.124 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:08.124 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:08.125 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:08.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.121+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef10005580 con 0x7fef28007190 2024-09-17T05:34:08.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.121+0000 7fef25ffb640 1 -- 172.21.15.117:0/184081892 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7fef2c06b6d0 con 0x7fef28007190 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.128+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef14078f90 msgr2=0x7fef1407b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.128+0000 7fef36358640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef14078f90 0x7fef1407b4a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef20036000 tx=0x7fef20004730 comp rx=0 tx=0).stop 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.128+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef28007190 msgr2=0x7fef280fecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.128+0000 7fef36358640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef28007190 0x7fef280fecd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00c450 tx=0x7fef2c00c910 comp rx=0 tx=0).stop 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 shutdown_connections 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef14078f90 0x7fef1407b4a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef280ffda0 0x7fef2819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef2804c180 0x7fef2819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 --2- 172.21.15.117:0/184081892 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef28007190 0x7fef280fecd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 >> 172.21.15.117:0/184081892 conn(0x7fef2804fe30 msgr2=0x7fef280ecbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:08.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.130+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 shutdown_connections 2024-09-17T05:34:08.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:08.131+0000 7fef36358640 1 -- 172.21.15.117:0/184081892 wait complete. 2024-09-17T05:34:08.150 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:10.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:10.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.348+0000 7fb76de3f640 1 -- 172.21.15.117:0/1503688258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7680c9e60 msgr2=0x7fb7680c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:10.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.348+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1503688258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7680c9e60 0x7fb7680c3540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009830 tx=0x7fb75c02a520 comp rx=0 tx=0).stop 2024-09-17T05:34:10.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.348+0000 7fb76de3f640 1 -- 172.21.15.117:0/1503688258 shutdown_connections 2024-09-17T05:34:10.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.348+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1503688258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7680c9e60 0x7fb7680c3540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.348+0000 7fb76de3f640 1 -- 172.21.15.117:0/1503688258 >> 172.21.15.117:0/1503688258 conn(0x7fb76806cda0 msgr2=0x7fb76806d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:10.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.349+0000 7fb76de3f640 1 -- 172.21.15.117:0/1503688258 shutdown_connections 2024-09-17T05:34:10.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.349+0000 7fb76de3f640 1 -- 172.21.15.117:0/1503688258 wait complete. 2024-09-17T05:34:10.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.353+0000 7fb76de3f640 1 Processor -- start 2024-09-17T05:34:10.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.353+0000 7fb76de3f640 1 -- start start 2024-09-17T05:34:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.355+0000 7fb76de3f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb7680c9e60 0x7fb7681df080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:10.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb76de3f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7681530e0 0x7fb7680c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:10.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb76de3f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7681574f0 0x7fb7680cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:10.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb76de3f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680c3a80 con 0x7fb7681530e0 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb76de3f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb768159540 con 0x7fb7681574f0 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb76de3f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb7680cad80 con 0x7fb7680c9e60 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb766ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7681530e0 0x7fb7680c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb766ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7681530e0 0x7fb7680c7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56536/0 (socket says 172.21.15.117:56536) 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb766ffd640 1 -- 172.21.15.117:0/1860859763 learned_addr learned my addr 172.21.15.117:0/1860859763 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb7677fe640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb7680c9e60 0x7fb7681df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb7677fe640 1 -- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7681574f0 msgr2=0x7fb7680cdf40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:34:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb7677fe640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7681574f0 0x7fb7680cdf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb7677fe640 1 -- 172.21.15.117:0/1860859763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7681530e0 msgr2=0x7fb7680c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb7677fe640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7681530e0 0x7fb7680c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.356+0000 7fb7677fe640 1 -- 172.21.15.117:0/1860859763 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7680c8aa0 con 0x7fb7680c9e60 2024-09-17T05:34:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.359+0000 7fb7677fe640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb7680c9e60 0x7fb7681df080 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb75800c3f0 tx=0x7fb75800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.359+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758011790 con 0x7fb7680c9e60 2024-09-17T05:34:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.360+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7681f2300 con 0x7fb7680c9e60 2024-09-17T05:34:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.360+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb768157280 con 0x7fb7680c9e60 2024-09-17T05:34:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.360+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb758017070 con 0x7fb7680c9e60 2024-09-17T05:34:10.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.360+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758011ce0 con 0x7fb7680c9e60 2024-09-17T05:34:10.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.365+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb748005290 con 0x7fb7680c9e60 2024-09-17T05:34:10.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.365+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb75801c460 con 0x7fb7680c9e60 2024-09-17T05:34:10.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.366+0000 7fb74ffff640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078bc0 0x7fb75407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:10.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.366+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fb75809ac20 con 0x7fb7680c9e60 2024-09-17T05:34:10.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.366+0000 7fb766ffd640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078bc0 0x7fb75407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:10.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.370+0000 7fb766ffd640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078bc0 0x7fb75407b0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb768158c40 tx=0x7fb75c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:10.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.376+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb758062c90 con 0x7fb7680c9e60 2024-09-17T05:34:10.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.776+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb748005580 con 0x7fb7680c9e60 2024-09-17T05:34:10.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.778+0000 7fb74ffff640 1 -- 172.21.15.117:0/1860859763 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7fb758062590 con 0x7fb7680c9e60 2024-09-17T05:34:10.779 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:10.779 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:10.779 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078bc0 msgr2=0x7fb75407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078bc0 0x7fb75407b0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb768158c40 tx=0x7fb75c004730 comp rx=0 tx=0).stop 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb7680c9e60 msgr2=0x7fb7681df080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb7680c9e60 0x7fb7681df080 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb75800c3f0 tx=0x7fb75800c8b0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 shutdown_connections 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb754078bc0 0x7fb75407b0d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7681574f0 0x7fb7680cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7681530e0 0x7fb7680c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 --2- 172.21.15.117:0/1860859763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb7680c9e60 0x7fb7681df080 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:10.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.781+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 >> 172.21.15.117:0/1860859763 conn(0x7fb76806cda0 msgr2=0x7fb7680cb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:10.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.782+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 shutdown_connections 2024-09-17T05:34:10.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:10.782+0000 7fb76de3f640 1 -- 172.21.15.117:0/1860859763 wait complete. 2024-09-17T05:34:10.799 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:12.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:12.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.994+0000 7f68e67fc640 1 -- 172.21.15.117:0/2296517941 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e0036400 con 0x7f68e8072bc0 2024-09-17T05:34:12.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.995+0000 7f68ec9de640 1 -- 172.21.15.117:0/2296517941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e8072bc0 msgr2=0x7f68e806df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:12.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.995+0000 7f68ec9de640 1 --2- 172.21.15.117:0/2296517941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e8072bc0 0x7f68e806df40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f68e806f790 tx=0x7f68e002aac0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.998+0000 7f68ec9de640 1 -- 172.21.15.117:0/2296517941 shutdown_connections 2024-09-17T05:34:13.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.998+0000 7f68ec9de640 1 --2- 172.21.15.117:0/2296517941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e8072bc0 0x7f68e806df40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.998+0000 7f68ec9de640 1 --2- 172.21.15.117:0/2296517941 >> v2:172.21.15.119:3300/0 conn(0x7f68e80c6b10 0x7f68e814db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.998+0000 7f68ec9de640 1 -- 172.21.15.117:0/2296517941 >> 172.21.15.117:0/2296517941 conn(0x7f68e80c8d10 msgr2=0x7f68e80c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:13.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.999+0000 7f68ec9de640 1 -- 172.21.15.117:0/2296517941 shutdown_connections 2024-09-17T05:34:13.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:12.999+0000 7f68ec9de640 1 -- 172.21.15.117:0/2296517941 wait complete. 2024-09-17T05:34:13.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.002+0000 7f68ec9de640 1 Processor -- start 2024-09-17T05:34:13.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.002+0000 7f68ec9de640 1 -- start start 2024-09-17T05:34:13.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68ec9de640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 0x7f68e81f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68ec9de640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e8072bc0 0x7f68e8155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68ec9de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e80c6b10 0x7f68e8146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68ec9de640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f68e8151860 con 0x7f68e806e480 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68ec9de640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f68e814e040 con 0x7f68e80c6b10 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68ec9de640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f68e8074780 con 0x7f68e8072bc0 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68e77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 0x7f68e81f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.004+0000 7f68e77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 0x7f68e81f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56568/0 (socket says 172.21.15.117:56568) 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e77fe640 1 -- 172.21.15.117:0/3212101491 learned_addr learned my addr 172.21.15.117:0/3212101491 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e6ffd640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e8072bc0 0x7f68e8155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e7fff640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e80c6b10 0x7f68e8146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e77fe640 1 -- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e8072bc0 msgr2=0x7f68e8155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:13.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e77fe640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e8072bc0 0x7f68e8155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e77fe640 1 -- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e80c6b10 msgr2=0x7f68e8146860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e77fe640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e80c6b10 0x7f68e8146860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.005+0000 7f68e77fe640 1 -- 172.21.15.117:0/3212101491 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e81f6d50 con 0x7f68e806e480 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.006+0000 7f68e77fe640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 0x7f68e81f6810 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f68d800b7b0 tx=0x7f68d800bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.006+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d8019070 con 0x7f68e806e480 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.006+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e8155c70 con 0x7f68e806e480 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.008+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68d800c810 con 0x7f68e806e480 2024-09-17T05:34:13.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.010+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d800cc60 con 0x7f68e806e480 2024-09-17T05:34:13.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.011+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f68d801c070 con 0x7f68e806e480 2024-09-17T05:34:13.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.012+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e814c960 con 0x7f68e806e480 2024-09-17T05:34:13.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.013+0000 7f68cffff640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68d4078be0 0x7f68d407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:13.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.014+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e8155e40 con 0x7f68e806e480 2024-09-17T05:34:13.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.017+0000 7f68e6ffd640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68d4078be0 0x7f68d407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:13.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.018+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f68d809a0b0 con 0x7f68e806e480 2024-09-17T05:34:13.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.027+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f68e81f6d50 con 0x7f68e806e480 2024-09-17T05:34:13.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.028+0000 7f68e6ffd640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68d4078be0 0x7f68d407b0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f68e0035000 tx=0x7f68e0004ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:13.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.403+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68e806f160 con 0x7f68e806e480 2024-09-17T05:34:13.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.404+0000 7f68cffff640 1 -- 172.21.15.117:0/3212101491 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f68d8061d60 con 0x7f68e806e480 2024-09-17T05:34:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:13.404 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:13.405 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68d4078be0 msgr2=0x7f68d407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68d4078be0 0x7f68d407b0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f68e0035000 tx=0x7f68e0004ab0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 msgr2=0x7f68e81f6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 0x7f68e81f6810 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f68d800b7b0 tx=0x7f68d800bc70 comp rx=0 tx=0).stop 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 shutdown_connections 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f68d4078be0 0x7f68d407b0f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68e80c6b10 0x7f68e8146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f68e8072bc0 0x7f68e8155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 --2- 172.21.15.117:0/3212101491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68e806e480 0x7f68e81f6810 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:13.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 >> 172.21.15.117:0/3212101491 conn(0x7f68e80c8d10 msgr2=0x7f68e80cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:13.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.406+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 shutdown_connections 2024-09-17T05:34:13.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:13.407+0000 7f68ec9de640 1 -- 172.21.15.117:0/3212101491 wait complete. 2024-09-17T05:34:13.421 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:15.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:15.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.630+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/1974159589 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 msgr2=0x7f8b940c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.630+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/1974159589 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 0x7f8b940c5660 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8b9415eae0 tx=0x7f8b8802a4b0 comp rx=0 tx=0).stop 2024-09-17T05:34:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.633+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/1974159589 shutdown_connections 2024-09-17T05:34:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.633+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/1974159589 >> v2:172.21.15.119:3300/0 conn(0x7f8b9406cda0 0x7f8b9406d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.633+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/1974159589 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 0x7f8b940c5660 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.633+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/1974159589 >> 172.21.15.117:0/1974159589 conn(0x7f8b9406e590 msgr2=0x7f8b940cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.635+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/1974159589 shutdown_connections 2024-09-17T05:34:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.639+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/1974159589 wait complete. 2024-09-17T05:34:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.641+0000 7f8b9a6b6640 1 Processor -- start 2024-09-17T05:34:15.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.642+0000 7f8b9a6b6640 1 -- start start 2024-09-17T05:34:15.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b9a6b6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b9406cda0 0x7f8b94152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b9a6b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 0x7f8b940cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b9a6b6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 0x7f8b94154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b9a6b6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9415ee90 con 0x7f8b94073dd0 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b9a6b6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9415ebb0 con 0x7f8b9406cda0 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b9a6b6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9415f170 con 0x7f8b94151fa0 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b98c53640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 0x7f8b94154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b98c53640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 0x7f8b94154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:58854/0 (socket says 172.21.15.117:58854) 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.644+0000 7f8b98c53640 1 -- 172.21.15.117:0/3864501472 learned_addr learned my addr 172.21.15.117:0/3864501472 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b937fe640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 0x7f8b940cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b93fff640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b9406cda0 0x7f8b94152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:15.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b98c53640 1 -- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b9406cda0 msgr2=0x7f8b94152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:15.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b98c53640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b9406cda0 0x7f8b94152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:15.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b98c53640 1 -- 172.21.15.117:0/3864501472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 msgr2=0x7f8b940cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:15.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b98c53640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 0x7f8b940cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:15.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.645+0000 7f8b98c53640 1 -- 172.21.15.117:0/3864501472 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b940cd540 con 0x7f8b94151fa0 2024-09-17T05:34:15.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.646+0000 7f8b98c53640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 0x7f8b94154fb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00ea70 tx=0x7f8b8c00ef30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:15.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.647+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00ce40 con 0x7f8b94151fa0 2024-09-17T05:34:15.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.648+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b8c004c10 con 0x7f8b94151fa0 2024-09-17T05:34:15.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.648+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b941fe890 con 0x7f8b94151fa0 2024-09-17T05:34:15.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.648+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b941fee50 con 0x7f8b94151fa0 2024-09-17T05:34:15.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.648+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c005020 con 0x7f8b94151fa0 2024-09-17T05:34:15.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.651+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b74005290 con 0x7f8b94151fa0 2024-09-17T05:34:15.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.653+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8b8c0051c0 con 0x7f8b94151fa0 2024-09-17T05:34:15.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.653+0000 7f8b90ff9640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8b80078c90 0x7f8b8007b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:15.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.654+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f8b8c09bda0 con 0x7f8b94151fa0 2024-09-17T05:34:15.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.654+0000 7f8b93fff640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8b80078c90 0x7f8b8007b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:15.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.655+0000 7f8b93fff640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8b80078c90 0x7f8b8007b1a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b88033000 tx=0x7f8b88004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:15.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:15.662+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8b8c063d70 con 0x7f8b94151fa0 2024-09-17T05:34:16.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.107+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b74005580 con 0x7f8b94151fa0 2024-09-17T05:34:16.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.112+0000 7f8b90ff9640 1 -- 172.21.15.117:0/3864501472 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f8b8c063670 con 0x7f8b94151fa0 2024-09-17T05:34:16.113 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:16.114 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:16.114 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:16.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.116+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8b80078c90 msgr2=0x7f8b8007b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:16.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8b80078c90 0x7f8b8007b1a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b88033000 tx=0x7f8b88004730 comp rx=0 tx=0).stop 2024-09-17T05:34:16.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 msgr2=0x7f8b94154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:16.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 0x7f8b94154fb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00ea70 tx=0x7f8b8c00ef30 comp rx=0 tx=0).stop 2024-09-17T05:34:16.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 shutdown_connections 2024-09-17T05:34:16.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8b80078c90 0x7f8b8007b1a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:16.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b94151fa0 0x7f8b94154fb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:16.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b94073dd0 0x7f8b940cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:16.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 --2- 172.21.15.117:0/3864501472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b9406cda0 0x7f8b94152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:16.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 >> 172.21.15.117:0/3864501472 conn(0x7f8b9406e590 msgr2=0x7f8b94151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:16.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 shutdown_connections 2024-09-17T05:34:16.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:16.117+0000 7f8b9a6b6640 1 -- 172.21.15.117:0/3864501472 wait complete. 2024-09-17T05:34:16.137 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-09-17T05:34:16.137 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:34:01:978270+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 46 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:34:01.978252+0000", "qdb_cluster": [ 4440, 5232 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:34:18.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:18.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.317+0000 7f4fc1d9b640 1 -- 172.21.15.117:0/156185040 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0047a0 con 0x7f4fc415bd10 2024-09-17T05:34:18.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/156185040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 msgr2=0x7f4fc415f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 --2- 172.21.15.117:0/156185040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc415f120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4fac005fd0 tx=0x7f4fac02a6a0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/156185040 shutdown_connections 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 --2- 172.21.15.117:0/156185040 >> v2:172.21.15.119:3300/0 conn(0x7f4fc40c92c0 0x7f4fc40c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 --2- 172.21.15.117:0/156185040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc415f120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/156185040 >> 172.21.15.117:0/156185040 conn(0x7f4fc406d190 msgr2=0x7f4fc406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/156185040 shutdown_connections 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.318+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/156185040 wait complete. 2024-09-17T05:34:18.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.321+0000 7f4fc8d4a640 1 Processor -- start 2024-09-17T05:34:18.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.321+0000 7f4fc8d4a640 1 -- start start 2024-09-17T05:34:18.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.323+0000 7f4fc8d4a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 0x7f4fc41f7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:18.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.324+0000 7f4fc259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 0x7f4fc41f7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:18.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.324+0000 7f4fc259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 0x7f4fc41f7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51816/0 (socket says 172.21.15.117:51816) 2024-09-17T05:34:18.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.324+0000 7f4fc259c640 1 -- 172.21.15.117:0/1917208217 learned_addr learned my addr 172.21.15.117:0/1917208217 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:18.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.324+0000 7f4fc8d4a640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc40c92c0 0x7f4fc40c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:18.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.324+0000 7f4fc8d4a640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc40c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.325+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc415c120 con 0x7f4fc4070890 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.325+0000 7f4fbbfff640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc40c92c0 0x7f4fc40c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.325+0000 7f4fc2d9d640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc40c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.325+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4152090 con 0x7f4fc40c92c0 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.325+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc4072e10 con 0x7f4fc415bd10 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc259c640 1 -- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 msgr2=0x7f4fc40c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:18.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc259c640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc40c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc259c640 1 -- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc40c92c0 msgr2=0x7f4fc40c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:18.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc259c640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc40c92c0 0x7f4fc40c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc259c640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc41f8400 con 0x7f4fc4070890 2024-09-17T05:34:18.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc2d9d640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc40c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:34:18.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fbbfff640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc40c92c0 0x7f4fc40c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:34:18.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.326+0000 7f4fc259c640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 0x7f4fc41f7ec0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009850 tx=0x7f4fac004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:18.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.328+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac03e030 con 0x7f4fc4070890 2024-09-17T05:34:18.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.328+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc40c3e10 con 0x7f4fc4070890 2024-09-17T05:34:18.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.328+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fac004e50 con 0x7f4fc4070890 2024-09-17T05:34:18.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.329+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac035040 con 0x7f4fc4070890 2024-09-17T05:34:18.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.331+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc40c35f0 con 0x7f4fc4070890 2024-09-17T05:34:18.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.332+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4fac041030 con 0x7f4fc4070890 2024-09-17T05:34:18.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.332+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc4151150 con 0x7f4fc4070890 2024-09-17T05:34:18.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.333+0000 7f4fbaffd640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4fa4078be0 0x7f4fa407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:18.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.335+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f4fac0b8e10 con 0x7f4fc4070890 2024-09-17T05:34:18.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.339+0000 7f4fbbfff640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4fa4078be0 0x7f4fa407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:18.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.342+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4fc4151150 con 0x7f4fc4070890 2024-09-17T05:34:18.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.344+0000 7f4fbbfff640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4fa4078be0 0x7f4fa407b0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40738b0 tx=0x7f4fb4005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:18.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.837+0000 7f4fc8d4a640 1 -- 172.21.15.117:0/1917208217 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fc4155850 con 0x7f4fc4070890 2024-09-17T05:34:18.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.839+0000 7f4fbaffd640 1 -- 172.21.15.117:0/1917208217 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f4fac0bc020 con 0x7f4fc4070890 2024-09-17T05:34:18.840 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:18.840 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:18.840 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:18.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 -- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4fa4078be0 msgr2=0x7f4fa407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:18.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4fa4078be0 0x7f4fa407b0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40738b0 tx=0x7f4fb4005f90 comp rx=0 tx=0).stop 2024-09-17T05:34:18.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 -- 172.21.15.117:0/1917208217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 msgr2=0x7f4fc41f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:18.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 0x7f4fc41f7ec0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009850 tx=0x7f4fac004a50 comp rx=0 tx=0).stop 2024-09-17T05:34:18.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 -- 172.21.15.117:0/1917208217 shutdown_connections 2024-09-17T05:34:18.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4fa4078be0 0x7f4fa407b0f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4fc415bd10 0x7f4fc40c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc40c92c0 0x7f4fc40c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:18.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 --2- 172.21.15.117:0/1917208217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fc4070890 0x7f4fc41f7ec0 secure :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009850 tx=0x7f4fac004a50 comp rx=0 tx=0).stop 2024-09-17T05:34:18.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.845+0000 7f4fb8ff9640 1 -- 172.21.15.117:0/1917208217 >> 172.21.15.117:0/1917208217 conn(0x7f4fc406d190 msgr2=0x7f4fc4068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:18.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.849+0000 7f4fb8ff9640 1 -- 172.21.15.117:0/1917208217 shutdown_connections 2024-09-17T05:34:18.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:18.849+0000 7f4fb8ff9640 1 -- 172.21.15.117:0/1917208217 wait complete. 2024-09-17T05:34:18.875 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:20.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:21.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.087+0000 7f21d1750640 1 -- 172.21.15.117:0/2748406494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 msgr2=0x7f21b400bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:21.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.087+0000 7f21c9ffb640 1 -- 172.21.15.117:0/2748406494 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc00c260 con 0x7f21b40099b0 2024-09-17T05:34:21.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.088+0000 7f21d1750640 1 --2- 172.21.15.117:0/2748406494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21b400bdd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00c6e0 tx=0x7f21b4005830 comp rx=0 tx=0).stop 2024-09-17T05:34:21.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.088+0000 7f21d1750640 1 -- 172.21.15.117:0/2748406494 shutdown_connections 2024-09-17T05:34:21.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.088+0000 7f21d1750640 1 --2- 172.21.15.117:0/2748406494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21b400bdd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.088+0000 7f21d1750640 1 -- 172.21.15.117:0/2748406494 >> 172.21.15.117:0/2748406494 conn(0x7f21cc06d190 msgr2=0x7f21cc06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:21.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.088+0000 7f21d1750640 1 -- 172.21.15.117:0/2748406494 shutdown_connections 2024-09-17T05:34:21.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.088+0000 7f21d1750640 1 -- 172.21.15.117:0/2748406494 wait complete. 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.092+0000 7f21d1750640 1 Processor -- start 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.092+0000 7f21d1750640 1 -- start start 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21d1750640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21cc1542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21caffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21cc1542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21caffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21cc1542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51852/0 (socket says 172.21.15.117:51852) 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21d1750640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21cc14bdb0 0x7f21cc14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21d1750640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21cc1526d0 0x7f21cc152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21d1750640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc15c120 con 0x7f21b40099b0 2024-09-17T05:34:21.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21d1750640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc15ec90 con 0x7f21cc1526d0 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21d1750640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f21cc072e10 con 0x7f21cc14bdb0 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.094+0000 7f21caffd640 1 -- 172.21.15.117:0/332807772 learned_addr learned my addr 172.21.15.117:0/332807772 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.095+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21cc14bdb0 0x7f21cc14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.095+0000 7f21caffd640 1 -- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21cc14bdb0 msgr2=0x7f21cc14c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.095+0000 7f21caffd640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21cc14bdb0 0x7f21cc14c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.095+0000 7f21caffd640 1 -- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21cc1526d0 msgr2=0x7f21cc152ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.095+0000 7f21caffd640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21cc1526d0 0x7f21cc152ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.095+0000 7f21caffd640 1 -- 172.21.15.117:0/332807772 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc1547f0 con 0x7f21b40099b0 2024-09-17T05:34:21.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.096+0000 7f21caffd640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21cc1542b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f21c400c7e0 tx=0x7f21c400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:21.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.096+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4017070 con 0x7f21b40099b0 2024-09-17T05:34:21.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.096+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21c4005180 con 0x7f21b40099b0 2024-09-17T05:34:21.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.096+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c400a640 con 0x7f21b40099b0 2024-09-17T05:34:21.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.096+0000 7f21d1750640 1 -- 172.21.15.117:0/332807772 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21cc0cc4a0 con 0x7f21b40099b0 2024-09-17T05:34:21.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.096+0000 7f21d1750640 1 -- 172.21.15.117:0/332807772 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21cc0cca60 con 0x7f21b40099b0 2024-09-17T05:34:21.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.098+0000 7f21d1750640 1 -- 172.21.15.117:0/332807772 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21cc155850 con 0x7f21b40099b0 2024-09-17T05:34:21.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.110+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f21c401a070 con 0x7f21b40099b0 2024-09-17T05:34:21.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.111+0000 7f21b37fe640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f21b8078c60 0x7f21b807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:21.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.111+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f21c409a860 con 0x7f21b40099b0 2024-09-17T05:34:21.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.111+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f21c4021020 con 0x7f21b40099b0 2024-09-17T05:34:21.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.115+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f21b8078c60 0x7f21b807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:21.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.119+0000 7f21ca7fc640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f21b8078c60 0x7f21b807b170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f21cc163b30 tx=0x7f21c0009530 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:21.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.548+0000 7f21d1750640 1 -- 172.21.15.117:0/332807772 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21cc0c6850 con 0x7f21b40099b0 2024-09-17T05:34:21.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.550+0000 7f21b37fe640 1 -- 172.21.15.117:0/332807772 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f21c4062270 con 0x7f21b40099b0 2024-09-17T05:34:21.553 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:21.553 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.556+0000 7f21b17fa640 1 -- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f21b8078c60 msgr2=0x7f21b807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.556+0000 7f21b17fa640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f21b8078c60 0x7f21b807b170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f21cc163b30 tx=0x7f21c0009530 comp rx=0 tx=0).stop 2024-09-17T05:34:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.556+0000 7f21b17fa640 1 -- 172.21.15.117:0/332807772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 msgr2=0x7f21cc1542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.556+0000 7f21b17fa640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21cc1542b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f21c400c7e0 tx=0x7f21c400cca0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 -- 172.21.15.117:0/332807772 shutdown_connections 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f21b8078c60 0x7f21b807b170 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f21cc1526d0 0x7f21cc152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f21cc14bdb0 0x7f21cc14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 --2- 172.21.15.117:0/332807772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21b40099b0 0x7f21cc1542b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 -- 172.21.15.117:0/332807772 >> 172.21.15.117:0/332807772 conn(0x7f21cc06d190 msgr2=0x7f21cc0be460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 -- 172.21.15.117:0/332807772 shutdown_connections 2024-09-17T05:34:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:21.557+0000 7f21b17fa640 1 -- 172.21.15.117:0/332807772 wait complete. 2024-09-17T05:34:21.581 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:23.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:23.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.789+0000 7f89a998b640 1 -- 172.21.15.117:0/1050344905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40d08f0 msgr2=0x7f89a4151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:23.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.789+0000 7f89a998b640 1 --2- 172.21.15.117:0/1050344905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40d08f0 0x7f89a4151da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89a40763a0 tx=0x7f899802a490 comp rx=0 tx=0).stop 2024-09-17T05:34:23.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.792+0000 7f89a998b640 1 -- 172.21.15.117:0/1050344905 shutdown_connections 2024-09-17T05:34:23.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.792+0000 7f89a998b640 1 --2- 172.21.15.117:0/1050344905 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f89a40d0d00 0x7f89a40c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:23.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.792+0000 7f89a998b640 1 --2- 172.21.15.117:0/1050344905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40d08f0 0x7f89a4151da0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:23.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.792+0000 7f89a998b640 1 --2- 172.21.15.117:0/1050344905 >> v2:172.21.15.117:3300/0 conn(0x7f89a40cb390 0x7f89a415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:23.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.792+0000 7f89a998b640 1 -- 172.21.15.117:0/1050344905 >> 172.21.15.117:0/1050344905 conn(0x7f89a4075e00 msgr2=0x7f89a406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:23.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.793+0000 7f89a998b640 1 -- 172.21.15.117:0/1050344905 shutdown_connections 2024-09-17T05:34:23.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.797+0000 7f89a998b640 1 -- 172.21.15.117:0/1050344905 wait complete. 2024-09-17T05:34:23.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.800+0000 7f89a998b640 1 Processor -- start 2024-09-17T05:34:23.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.800+0000 7f89a998b640 1 -- start start 2024-09-17T05:34:23.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.802+0000 7f89a998b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40cb390 0x7f89a415c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.802+0000 7f89a998b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 0x7f89a40cf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 0x7f89a40cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 0x7f89a40cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51862/0 (socket says 172.21.15.117:51862) 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 -- 172.21.15.117:0/3234878573 learned_addr learned my addr 172.21.15.117:0/3234878573 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a998b640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f89a40d0d00 0x7f89a4156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a998b640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89a415f690 con 0x7f89a40d08f0 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a998b640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f89a415f3b0 con 0x7f89a40cb390 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a998b640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f89a40d1110 con 0x7f89a40d0d00 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a37fe640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f89a40d0d00 0x7f89a4156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a2ffd640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40cb390 0x7f89a415c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 -- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f89a40d0d00 msgr2=0x7f89a4156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f89a40d0d00 0x7f89a4156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:23.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 -- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40cb390 msgr2=0x7f89a415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40cb390 0x7f89a415c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.803+0000 7f89a27fc640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a408cdd0 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.804+0000 7f89a27fc640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 0x7f89a40cf800 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8998009780 tx=0x7f8998033f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.806+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998031c10 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.806+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8998005020 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.807+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998005430 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.808+0000 7f89a998b640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a4155c40 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.811+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f89980047a0 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.811+0000 7f898bfff640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8990078c10 0x7f899007b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.812+0000 7f89a998b640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a414c440 con 0x7f89a40d08f0 2024-09-17T05:34:23.815 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.812+0000 7f89a2ffd640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8990078c10 0x7f899007b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:23.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.814+0000 7f89a2ffd640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8990078c10 0x7f899007b120 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8994005fd0 tx=0x7f8994009260 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:23.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.814+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f89980a7510 con 0x7f89a40d08f0 2024-09-17T05:34:23.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.816+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a40c7950 con 0x7f89a40d08f0 2024-09-17T05:34:23.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:23.825+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f899803f030 con 0x7f89a40d08f0 2024-09-17T05:34:24.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.262+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89a40cb810 con 0x7f89a40d08f0 2024-09-17T05:34:24.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.264+0000 7f898bfff640 1 -- 172.21.15.117:0/3234878573 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f8998083a90 con 0x7f89a40d08f0 2024-09-17T05:34:24.268 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:24.268 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:24.268 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:24.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8990078c10 msgr2=0x7f899007b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:24.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8990078c10 0x7f899007b120 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8994005fd0 tx=0x7f8994009260 comp rx=0 tx=0).stop 2024-09-17T05:34:24.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 msgr2=0x7f89a40cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:24.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 0x7f89a40cf800 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8998009780 tx=0x7f8998033f10 comp rx=0 tx=0).stop 2024-09-17T05:34:24.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 shutdown_connections 2024-09-17T05:34:24.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8990078c10 0x7f899007b120 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:24.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f89a40d0d00 0x7f89a4156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:24.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a40d08f0 0x7f89a40cf800 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:24.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 --2- 172.21.15.117:0/3234878573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89a40cb390 0x7f89a415c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:24.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.270+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 >> 172.21.15.117:0/3234878573 conn(0x7f89a4075e00 msgr2=0x7f89a4074d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:24.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.271+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 shutdown_connections 2024-09-17T05:34:24.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:24.271+0000 7f8989ffb640 1 -- 172.21.15.117:0/3234878573 wait complete. 2024-09-17T05:34:24.297 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:26.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.502+0000 7f485ebba640 1 -- 172.21.15.117:0/16001344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4840003a00 msgr2=0x7f4840003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.502+0000 7f485ebba640 1 --2- 172.21.15.117:0/16001344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4840003a00 0x7f4840003e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f484800c310 tx=0x7f4840005830 comp rx=0 tx=0).stop 2024-09-17T05:34:26.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.502+0000 7f485d157640 1 -- 172.21.15.117:0/16001344 reap_dead start 2024-09-17T05:34:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.504+0000 7f485ebba640 1 -- 172.21.15.117:0/16001344 shutdown_connections 2024-09-17T05:34:26.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.504+0000 7f485ebba640 1 -- 172.21.15.117:0/16001344 >> 172.21.15.117:0/16001344 conn(0x7f485806d190 msgr2=0x7f485806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.504+0000 7f485ebba640 1 -- 172.21.15.117:0/16001344 shutdown_connections 2024-09-17T05:34:26.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.504+0000 7f485ebba640 1 -- 172.21.15.117:0/16001344 wait complete. 2024-09-17T05:34:26.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.509+0000 7f485ebba640 1 Processor -- start 2024-09-17T05:34:26.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.514+0000 7f485ebba640 1 -- start start 2024-09-17T05:34:26.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.517+0000 7f485ebba640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4840003a00 0x7f48580c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:26.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.518+0000 7f485ebba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581f13c0 0x7f48580c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:26.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.518+0000 7f485ebba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48581f0790 0x7f48581f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:26.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.519+0000 7f485ebba640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4858073f60 con 0x7f48581f13c0 2024-09-17T05:34:26.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.519+0000 7f485ebba640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4858074e30 con 0x7f48581f0790 2024-09-17T05:34:26.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.519+0000 7f485ebba640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f48580c4c10 con 0x7f4840003a00 2024-09-17T05:34:26.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.520+0000 7f485c956640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4840003a00 0x7f48580c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:26.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.521+0000 7f485c956640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4840003a00 0x7f48580c0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48438/0 (socket says 172.21.15.117:48438) 2024-09-17T05:34:26.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.521+0000 7f485c956640 1 -- 172.21.15.117:0/2003748801 learned_addr learned my addr 172.21.15.117:0/2003748801 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:26.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.521+0000 7f485d157640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48581f0790 0x7f48581f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.522+0000 7f4857fff640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581f13c0 0x7f48580c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.523+0000 7f4857fff640 1 -- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4840003a00 msgr2=0x7f48580c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.523+0000 7f4857fff640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4840003a00 0x7f48580c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.523+0000 7f4857fff640 1 -- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48581f0790 msgr2=0x7f48581f0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.523+0000 7f4857fff640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48581f0790 0x7f48581f0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.523+0000 7f4857fff640 1 -- 172.21.15.117:0/2003748801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4858149de0 con 0x7f48581f13c0 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.523+0000 7f4857fff640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581f13c0 0x7f48580c2e30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f484c00ae10 tx=0x7f484c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.524+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c00d9c0 con 0x7f48581f13c0 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.524+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4858152670 con 0x7f48581f13c0 2024-09-17T05:34:26.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.524+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f485814a180 con 0x7f48581f13c0 2024-09-17T05:34:26.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.524+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f484c00db60 con 0x7f48581f13c0 2024-09-17T05:34:26.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.524+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c018020 con 0x7f48581f13c0 2024-09-17T05:34:26.529 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.527+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f484c0181c0 con 0x7f48581f13c0 2024-09-17T05:34:26.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.529+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4838005290 con 0x7f48581f13c0 2024-09-17T05:34:26.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.539+0000 7f48557fa640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4844078b90 0x7f484407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:26.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.539+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f484c0a4520 con 0x7f48581f13c0 2024-09-17T05:34:26.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.539+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4858149de0 con 0x7f48581f13c0 2024-09-17T05:34:26.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.543+0000 7f485c956640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4844078b90 0x7f484407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:26.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.548+0000 7f485c956640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4844078b90 0x7f484407b0a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4848000c00 tx=0x7f4858158800 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:26.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.986+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4838005580 con 0x7f48581f13c0 2024-09-17T05:34:26.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.988+0000 7f48557fa640 1 -- 172.21.15.117:0/2003748801 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f484c06beb0 con 0x7f48581f13c0 2024-09-17T05:34:26.989 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:26.990 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:26.990 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:26.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4844078b90 msgr2=0x7f484407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:26.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4844078b90 0x7f484407b0a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4848000c00 tx=0x7f4858158800 comp rx=0 tx=0).stop 2024-09-17T05:34:26.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581f13c0 msgr2=0x7f48580c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:26.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581f13c0 0x7f48580c2e30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f484c00ae10 tx=0x7f484c00eec0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 shutdown_connections 2024-09-17T05:34:26.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4844078b90 0x7f484407b0a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48581f0790 0x7f48581f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581f13c0 0x7f48580c2e30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 --2- 172.21.15.117:0/2003748801 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4840003a00 0x7f48580c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:26.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.992+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 >> 172.21.15.117:0/2003748801 conn(0x7f485806d190 msgr2=0x7f485814b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:26.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.993+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 shutdown_connections 2024-09-17T05:34:26.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:26.993+0000 7f485ebba640 1 -- 172.21.15.117:0/2003748801 wait complete. 2024-09-17T05:34:27.014 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:29.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:29.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.245+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2257145371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 msgr2=0x7f3a080db6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:29.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.245+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2257145371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a080db6c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009830 tx=0x7f3a0002a4d0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.247+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2257145371 shutdown_connections 2024-09-17T05:34:29.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.247+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2257145371 >> v2:172.21.15.117:3300/0 conn(0x7f3a08044cb0 0x7f3a080d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.247+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2257145371 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a080db6c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.247+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2257145371 >> 172.21.15.117:0/2257145371 conn(0x7f3a0804fe30 msgr2=0x7f3a08050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:29.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.248+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2257145371 shutdown_connections 2024-09-17T05:34:29.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.248+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2257145371 wait complete. 2024-09-17T05:34:29.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.251+0000 7f3a0ffff640 1 Processor -- start 2024-09-17T05:34:29.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.252+0000 7f3a0ffff640 1 -- start start 2024-09-17T05:34:29.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.254+0000 7f3a0ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a08049f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:29.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.254+0000 7f3a0ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a08044cb0 0x7f3a08186010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:29.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.254+0000 7f3a0ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a080469d0 0x7f3a08182ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:29.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.254+0000 7f3a0ffff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a080461d0 con 0x7f3a080469d0 2024-09-17T05:34:29.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.254+0000 7f3a0ffff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a08045ef0 con 0x7f3a08044cb0 2024-09-17T05:34:29.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.254+0000 7f3a0ffff640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f3a080d5200 con 0x7f3a0800de50 2024-09-17T05:34:29.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.255+0000 7f3a0e7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a08044cb0 0x7f3a08186010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:29.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.256+0000 7f3a0effd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a08049f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:29.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.256+0000 7f3a0f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a080469d0 0x7f3a08182ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:29.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.256+0000 7f3a0effd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a08049f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48468/0 (socket says 172.21.15.117:48468) 2024-09-17T05:34:29.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.256+0000 7f3a0effd640 1 -- 172.21.15.117:0/2209045230 learned_addr learned my addr 172.21.15.117:0/2209045230 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:29.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.257+0000 7f3a0f7fe640 1 -- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 msgr2=0x7f3a08049f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:29.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.257+0000 7f3a0f7fe640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a08049f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.257+0000 7f3a0f7fe640 1 -- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a08044cb0 msgr2=0x7f3a08186010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:29.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.257+0000 7f3a0f7fe640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a08044cb0 0x7f3a08186010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.257+0000 7f3a0f7fe640 1 -- 172.21.15.117:0/2209045230 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0804a4a0 con 0x7f3a080469d0 2024-09-17T05:34:29.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.258+0000 7f3a0f7fe640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a080469d0 0x7f3a08182ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c750 tx=0x7f3a0400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:29.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.258+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04005030 con 0x7f3a080469d0 2024-09-17T05:34:29.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.258+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a040051f0 con 0x7f3a080469d0 2024-09-17T05:34:29.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.258+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a040046b0 con 0x7f3a080469d0 2024-09-17T05:34:29.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.260+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a08034b80 con 0x7f3a080469d0 2024-09-17T05:34:29.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.260+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a080482e0 con 0x7f3a080469d0 2024-09-17T05:34:29.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.261+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a080db6c0 con 0x7f3a080469d0 2024-09-17T05:34:29.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.272+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a04004850 con 0x7f3a080469d0 2024-09-17T05:34:29.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.273+0000 7f39f77fe640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39fc078b90 0x7f39fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:29.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.273+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f3a0409b800 con 0x7f3a080469d0 2024-09-17T05:34:29.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.273+0000 7f3a0effd640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39fc078b90 0x7f39fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:29.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.273+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a040cbe30 con 0x7f3a080469d0 2024-09-17T05:34:29.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.286+0000 7f3a0effd640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39fc078b90 0x7f39fc07b0a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a00000c00 tx=0x7f3a00004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:29.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.723+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a08032a10 con 0x7f3a080469d0 2024-09-17T05:34:29.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.724+0000 7f39f77fe640 1 -- 172.21.15.117:0/2209045230 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f3a040633a0 con 0x7f3a080469d0 2024-09-17T05:34:29.726 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:29.726 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:29.726 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:29.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39fc078b90 msgr2=0x7f39fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:29.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39fc078b90 0x7f39fc07b0a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a00000c00 tx=0x7f3a00004730 comp rx=0 tx=0).stop 2024-09-17T05:34:29.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a080469d0 msgr2=0x7f3a08182ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a080469d0 0x7f3a08182ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c750 tx=0x7f3a0400cc10 comp rx=0 tx=0).stop 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 shutdown_connections 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f39fc078b90 0x7f39fc07b0a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a080469d0 0x7f3a08182ff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a08044cb0 0x7f3a08186010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 --2- 172.21.15.117:0/2209045230 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a0800de50 0x7f3a08049f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.728+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 >> 172.21.15.117:0/2209045230 conn(0x7f3a0804fe30 msgr2=0x7f3a080342d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:29.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.729+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 shutdown_connections 2024-09-17T05:34:29.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:29.729+0000 7f3a0ffff640 1 -- 172.21.15.117:0/2209045230 wait complete. 2024-09-17T05:34:29.751 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-09-17T05:34:29.752 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:34:01:978270+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5082": { "addr": "172.21.15.117:6837/3188534361", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 3188534361, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 3188534361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5082, "incarnation": 21, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:stopping", "state_seq": 46 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:34:01.978252+0000", "qdb_cluster": [ 4440, 5232 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232, "mds_2": 5082 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:34:31.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:32.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.015+0000 7f1158113640 1 -- 172.21.15.117:0/1409005440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148044cb0 msgr2=0x7f11480db6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:32.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.015+0000 7f1158113640 1 --2- 172.21.15.117:0/1409005440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148044cb0 0x7f11480db6c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f11480466c0 tx=0x7f114c02a5d0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.016+0000 7f1158113640 1 -- 172.21.15.117:0/1409005440 shutdown_connections 2024-09-17T05:34:32.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.016+0000 7f1158113640 1 --2- 172.21.15.117:0/1409005440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f11480469d0 0x7f11480d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.016+0000 7f1158113640 1 --2- 172.21.15.117:0/1409005440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148044cb0 0x7f11480db6c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.016+0000 7f1158113640 1 -- 172.21.15.117:0/1409005440 >> 172.21.15.117:0/1409005440 conn(0x7f114804fe30 msgr2=0x7f1148050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.016+0000 7f1158113640 1 -- 172.21.15.117:0/1409005440 shutdown_connections 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.016+0000 7f1158113640 1 -- 172.21.15.117:0/1409005440 wait complete. 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.019+0000 7f1158113640 1 Processor -- start 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.019+0000 7f1158113640 1 -- start start 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.022+0000 7f1158113640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f114800de50 0x7f11480c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.022+0000 7f1158113640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 0x7f11480050e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.022+0000 7f1158113640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11480469d0 0x7f11480d4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f11556ae640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 0x7f11480050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f11556ae640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 0x7f11480050e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:48482/0 (socket says 172.21.15.117:48482) 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f11556ae640 1 -- 172.21.15.117:0/3546424041 learned_addr learned my addr 172.21.15.117:0/3546424041 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f1155eaf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f114800de50 0x7f11480c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:32.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f11566b0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11480469d0 0x7f11480d4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f1158113640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11480461d0 con 0x7f11480469d0 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f1158113640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1148045ef0 con 0x7f114800de50 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.023+0000 7f1158113640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f11480d5200 con 0x7f1148044cb0 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.024+0000 7f1155eaf640 1 -- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 msgr2=0x7f11480050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.024+0000 7f1155eaf640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 0x7f11480050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.024+0000 7f1155eaf640 1 -- 172.21.15.117:0/3546424041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11480469d0 msgr2=0x7f11480d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.024+0000 7f1155eaf640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11480469d0 0x7f11480d4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.024+0000 7f1155eaf640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1148178df0 con 0x7f114800de50 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.025+0000 7f11556ae640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 0x7f11480050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.025+0000 7f11566b0640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11480469d0 0x7f11480d4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.026+0000 7f1155eaf640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f114800de50 0x7f11480c9c50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f11500759d0 tx=0x7f115006a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.026+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150073630 con 0x7f114800de50 2024-09-17T05:34:32.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.027+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11500738e0 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.027+0000 7f1158113640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11480c6650 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.027+0000 7f1158113640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11480c6370 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.029+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150073d30 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.032+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1150095480 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.033+0000 7f11467fc640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1134078b90 0x7f113407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.033+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1130005290 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.033+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f1150117480 con 0x7f114800de50 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.035+0000 7f11556ae640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1134078b90 0x7f113407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.036+0000 7f11556ae640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1134078b90 0x7f113407b0a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f114c009230 tx=0x7f114c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:32.100 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.044+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f11500deea0 con 0x7f114800de50 2024-09-17T05:34:32.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.489+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1130005580 con 0x7f114800de50 2024-09-17T05:34:32.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.492+0000 7f11467fc640 1 -- 172.21.15.117:0/3546424041 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f11500e38d0 con 0x7f114800de50 2024-09-17T05:34:32.499 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:32.499 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:32.499 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:32.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.498+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1134078b90 msgr2=0x7f113407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:32.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.498+0000 7f113bfff640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1134078b90 0x7f113407b0a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f114c009230 tx=0x7f114c004730 comp rx=0 tx=0).stop 2024-09-17T05:34:32.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.498+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f114800de50 msgr2=0x7f11480c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:32.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.498+0000 7f113bfff640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f114800de50 0x7f11480c9c50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f11500759d0 tx=0x7f115006a0a0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 shutdown_connections 2024-09-17T05:34:32.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1134078b90 0x7f113407b0a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11480469d0 0x7f11480d4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1148044cb0 0x7f11480050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 --2- 172.21.15.117:0/3546424041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f114800de50 0x7f11480c9c50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:32.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 >> 172.21.15.117:0/3546424041 conn(0x7f114804fe30 msgr2=0x7f11480ca830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:32.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 shutdown_connections 2024-09-17T05:34:32.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:32.499+0000 7f113bfff640 1 -- 172.21.15.117:0/3546424041 wait complete. 2024-09-17T05:34:32.525 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:34.525 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:34.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.729+0000 7f88e7027640 1 -- 172.21.15.117:0/3079383784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 msgr2=0x7f88e0070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:34.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.729+0000 7f88e7027640 1 --2- 172.21.15.117:0/3079383784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e0070ca0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f88e00724d0 tx=0x7f88d802aaa0 comp rx=0 tx=0).stop 2024-09-17T05:34:34.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 -- 172.21.15.117:0/3079383784 shutdown_connections 2024-09-17T05:34:34.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 --2- 172.21.15.117:0/3079383784 >> v2:172.21.15.117:3300/0 conn(0x7f88e00c92c0 0x7f88e00c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:34.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 --2- 172.21.15.117:0/3079383784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e0070ca0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:34.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 --2- 172.21.15.117:0/3079383784 >> v2:172.21.15.119:3300/0 conn(0x7f88e015bd10 0x7f88e015f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:34.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 -- 172.21.15.117:0/3079383784 >> 172.21.15.117:0/3079383784 conn(0x7f88e006d190 msgr2=0x7f88e006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:34.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 -- 172.21.15.117:0/3079383784 shutdown_connections 2024-09-17T05:34:34.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.732+0000 7f88e7027640 1 -- 172.21.15.117:0/3079383784 wait complete. 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.735+0000 7f88e7027640 1 Processor -- start 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.735+0000 7f88e7027640 1 -- start start 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.737+0000 7f88e7027640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e00c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.737+0000 7f88e7027640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e00c92c0 0x7f88e00cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.737+0000 7f88e7027640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88e015bd10 0x7f88e00cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.737+0000 7f88e7027640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88e015c120 con 0x7f88e00c92c0 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.737+0000 7f88e7027640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f88e0152090 con 0x7f88e015bd10 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.737+0000 7f88e7027640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f88e0072e10 con 0x7f88e0070890 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e00c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e00c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:58310/0 (socket says 172.21.15.117:58310) 2024-09-17T05:34:34.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 -- 172.21.15.117:0/1726880431 learned_addr learned my addr 172.21.15.117:0/1726880431 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e55c4640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88e015bd10 0x7f88e00cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88dffff640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e00c92c0 0x7f88e00cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 -- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88e015bd10 msgr2=0x7f88e00cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88e015bd10 0x7f88e00cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 -- 172.21.15.117:0/1726880431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e00c92c0 msgr2=0x7f88e00cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e00c92c0 0x7f88e00cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88e4dc3640 1 -- 172.21.15.117:0/1726880431 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e00c3980 con 0x7f88e0070890 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.738+0000 7f88dffff640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e00c92c0 0x7f88e00cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.739+0000 7f88e4dc3640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e00c43c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88d400b410 tx=0x7f88d400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:34.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.739+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4015020 con 0x7f88e0070890 2024-09-17T05:34:34.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.739+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e008cd50 con 0x7f88e0070890 2024-09-17T05:34:34.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.739+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e015e560 con 0x7f88e0070890 2024-09-17T05:34:34.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.741+0000 7f88c6ffd640 1 -- 172.21.15.117:0/1726880431 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c0005290 con 0x7f88e0070890 2024-09-17T05:34:34.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.742+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88d4011920 con 0x7f88e0070890 2024-09-17T05:34:34.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.743+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4011d30 con 0x7f88e0070890 2024-09-17T05:34:34.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.743+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f88d4018020 con 0x7f88e0070890 2024-09-17T05:34:34.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.743+0000 7f88dd7fa640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f88cc078bc0 0x7f88cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:34.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.744+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f88d409b1d0 con 0x7f88e0070890 2024-09-17T05:34:34.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.744+0000 7f88dffff640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f88cc078bc0 0x7f88cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:34.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.745+0000 7f88dffff640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f88cc078bc0 0x7f88cc07b0d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88e015e010 tx=0x7f88d0005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:34.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:34.751+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f88d401f020 con 0x7f88e0070890 2024-09-17T05:34:35.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.201+0000 7f88c6ffd640 1 -- 172.21.15.117:0/1726880431 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88c0005580 con 0x7f88e0070890 2024-09-17T05:34:35.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.202+0000 7f88dd7fa640 1 -- 172.21.15.117:0/1726880431 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6926 (secure 0 0 0) 0x7f88d4063230 con 0x7f88e0070890 2024-09-17T05:34:35.205 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:35.205 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":34,"btime":"2024-09-17T05:34:01:978270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:01.978252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":4440,"mds_1":5232,"mds_2":5082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5082":{"gid":5082,"name":"e","rank":2,"incarnation":21,"state":"up:stopping","state_seq":46,"addr":"172.21.15.117:6837/3188534361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3188534361},{"type":"v1","addr":"172.21.15.117:6837","nonce":3188534361}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:active","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440,5232]},"id":1}]} 2024-09-17T05:34:35.205 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 34 2024-09-17T05:34:35.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.206+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f88cc078bc0 msgr2=0x7f88cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:35.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.206+0000 7f88e7027640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f88cc078bc0 0x7f88cc07b0d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88e015e010 tx=0x7f88d0005f90 comp rx=0 tx=0).stop 2024-09-17T05:34:35.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.207+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 msgr2=0x7f88e00c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:35.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.207+0000 7f88e7027640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e00c43c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88d400b410 tx=0x7f88d400b8d0 comp rx=0 tx=0).stop 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.209+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 shutdown_connections 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.209+0000 7f88e7027640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f88cc078bc0 0x7f88cc07b0d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.209+0000 7f88e7027640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88e015bd10 0x7f88e00cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.209+0000 7f88e7027640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e00c92c0 0x7f88e00cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.209+0000 7f88e7027640 1 --2- 172.21.15.117:0/1726880431 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88e0070890 0x7f88e00c43c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.209+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 >> 172.21.15.117:0/1726880431 conn(0x7f88e006d190 msgr2=0x7f88e0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:35.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.210+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 shutdown_connections 2024-09-17T05:34:35.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:35.210+0000 7f88e7027640 1 -- 172.21.15.117:0/1726880431 wait complete. 2024-09-17T05:34:35.236 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:37.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:37.268 INFO:tasks.ceph.mds.e.smithi117.stdout:starting mds.e at 2024-09-17T05:34:37.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.456+0000 7fa6394f5640 1 -- 172.21.15.117:0/2641239967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c0469d0 msgr2=0x7fa62c0d16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:37.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.456+0000 7fa6394f5640 1 --2- 172.21.15.117:0/2641239967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c0469d0 0x7fa62c0d16a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0466c0 tx=0x7fa62802a5c0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.457+0000 7fa6394f5640 1 -- 172.21.15.117:0/2641239967 shutdown_connections 2024-09-17T05:34:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.457+0000 7fa6394f5640 1 --2- 172.21.15.117:0/2641239967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c0469d0 0x7fa62c0d16a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.457+0000 7fa6394f5640 1 --2- 172.21.15.117:0/2641239967 >> v2:172.21.15.119:3300/0 conn(0x7fa62c00de50 0x7fa62c0db6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.457+0000 7fa6394f5640 1 -- 172.21.15.117:0/2641239967 >> 172.21.15.117:0/2641239967 conn(0x7fa62c04fe30 msgr2=0x7fa62c050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.458+0000 7fa6394f5640 1 -- 172.21.15.117:0/2641239967 shutdown_connections 2024-09-17T05:34:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.458+0000 7fa6394f5640 1 -- 172.21.15.117:0/2641239967 wait complete. 2024-09-17T05:34:37.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.462+0000 7fa6394f5640 1 Processor -- start 2024-09-17T05:34:37.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.462+0000 7fa6394f5640 1 -- start start 2024-09-17T05:34:37.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.464+0000 7fa6394f5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c00de50 0x7fa62c0d2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:37.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.466+0000 7fa633fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c00de50 0x7fa62c0d2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:37.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa633fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c00de50 0x7fa62c0d2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52202/0 (socket says 172.21.15.117:52202) 2024-09-17T05:34:37.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa633fff640 1 -- 172.21.15.117:0/1558647919 learned_addr learned my addr 172.21.15.117:0/1558647919 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:37.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa62c044cb0 0x7fa62c0d4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:37.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa62c0469d0 0x7fa62c0c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:37.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0461d0 con 0x7fa62c00de50 2024-09-17T05:34:37.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c045ef0 con 0x7fa62c0469d0 2024-09-17T05:34:37.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa638cf4640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa62c0469d0 0x7fa62c0c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:37.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.467+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c0d5200 con 0x7fa62c044cb0 2024-09-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.468+0000 7fa638cf4640 1 -- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa62c044cb0 msgr2=0x7fa62c0d4160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.468+0000 7fa638cf4640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa62c044cb0 0x7fa62c0d4160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.468+0000 7fa638cf4640 1 -- 172.21.15.117:0/1558647919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c00de50 msgr2=0x7fa62c0d2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.468+0000 7fa638cf4640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c00de50 0x7fa62c0d2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.468+0000 7fa638cf4640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c048290 con 0x7fa62c0469d0 2024-09-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.469+0000 7fa638cf4640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa62c0469d0 0x7fa62c0c3600 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa628004000 tx=0x7fa628003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:37.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.471+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628041020 con 0x7fa62c0469d0 2024-09-17T05:34:37.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.471+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa628037040 con 0x7fa62c0469d0 2024-09-17T05:34:37.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.471+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628036070 con 0x7fa62c0469d0 2024-09-17T05:34:37.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.472+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c047c00 con 0x7fa62c0469d0 2024-09-17T05:34:37.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.472+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1765c0 con 0x7fa62c0469d0 2024-09-17T05:34:37.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.473+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c0ca7e0 con 0x7fa62c0469d0 2024-09-17T05:34:37.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.476+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa628044020 con 0x7fa62c0469d0 2024-09-17T05:34:37.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.476+0000 7fa630ff9640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa618078c90 0x7fa61807b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:37.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.477+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fa6280bc290 con 0x7fa62c0469d0 2024-09-17T05:34:37.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.478+0000 7fa633fff640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa618078c90 0x7fa61807b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:37.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.479+0000 7fa633fff640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa618078c90 0x7fa61807b1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa6200099a0 tx=0x7fa620006f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:37.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.485+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa62c0ca7e0 con 0x7fa62c0469d0 2024-09-17T05:34:37.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.941+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa62c0db6c0 con 0x7fa62c0469d0 2024-09-17T05:34:37.947 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:37.947 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":36,"btime":"2024-09-17T05:34:37:169159+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:37.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.943+0000 7fa630ff9640 1 -- 172.21.15.117:0/1558647919 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6079 (secure 0 0 0) 0x7fa62804b030 con 0x7fa62c0469d0 2024-09-17T05:34:37.948 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 36 2024-09-17T05:34:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa618078c90 msgr2=0x7fa61807b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa618078c90 0x7fa61807b1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa6200099a0 tx=0x7fa620006f90 comp rx=0 tx=0).stop 2024-09-17T05:34:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa62c0469d0 msgr2=0x7fa62c0c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa62c0469d0 0x7fa62c0c3600 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa628004000 tx=0x7fa628003f90 comp rx=0 tx=0).stop 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 shutdown_connections 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa618078c90 0x7fa61807b1a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa62c0469d0 0x7fa62c0c3600 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa62c044cb0 0x7fa62c0d4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 --2- 172.21.15.117:0/1558647919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa62c00de50 0x7fa62c0d2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.948+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 >> 172.21.15.117:0/1558647919 conn(0x7fa62c04fe30 msgr2=0x7fa62c0dae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.949+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 shutdown_connections 2024-09-17T05:34:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:37.949+0000 7fa6394f5640 1 -- 172.21.15.117:0/1558647919 wait complete. 2024-09-17T05:34:37.983 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:39.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:40.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.179+0000 7f8306769640 1 -- 172.21.15.117:0/505035986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 msgr2=0x7f83000c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:40.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.179+0000 7f8306769640 1 --2- 172.21.15.117:0/505035986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83000c5b80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f83000c4c40 tx=0x7f82f802ec90 comp rx=0 tx=0).stop 2024-09-17T05:34:40.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.180+0000 7f8306769640 1 -- 172.21.15.117:0/505035986 shutdown_connections 2024-09-17T05:34:40.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.180+0000 7f8306769640 1 --2- 172.21.15.117:0/505035986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83000c5b80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.180+0000 7f8306769640 1 --2- 172.21.15.117:0/505035986 >> v2:172.21.15.119:3300/0 conn(0x7f830014dc50 0x7f8300157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.183 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.180+0000 7f8306769640 1 -- 172.21.15.117:0/505035986 >> 172.21.15.117:0/505035986 conn(0x7f830006f0e0 msgr2=0x7f8300070720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:40.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.184+0000 7f8306769640 1 -- 172.21.15.117:0/505035986 shutdown_connections 2024-09-17T05:34:40.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.185+0000 7f8306769640 1 -- 172.21.15.117:0/505035986 wait complete. 2024-09-17T05:34:40.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.188+0000 7f8306769640 1 Processor -- start 2024-09-17T05:34:40.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.188+0000 7f8306769640 1 -- start start 2024-09-17T05:34:40.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.191+0000 7f8306769640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8300072b30 0x7f83001e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:40.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.193+0000 7f8306769640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83001fed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:40.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.193+0000 7f8306769640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f830014dc50 0x7f83001ed330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:40.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.193+0000 7f8306769640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f83000c60c0 con 0x7f83000c5770 2024-09-17T05:34:40.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.193+0000 7f8306769640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f83000739c0 con 0x7f830014dc50 2024-09-17T05:34:40.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f8306769640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f83000c6d60 con 0x7f8300072b30 2024-09-17T05:34:40.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83001fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83001fed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52220/0 (socket says 172.21.15.117:52220) 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 -- 172.21.15.117:0/785040725 learned_addr learned my addr 172.21.15.117:0/785040725 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f8304d06640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f830014dc50 0x7f83001ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 -- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8300072b30 msgr2=0x7f83001e08c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8300072b30 0x7f83001e08c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 -- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f830014dc50 msgr2=0x7f83001ed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f830014dc50 0x7f83001ed330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.194+0000 7f82ff7fe640 1 -- 172.21.15.117:0/785040725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83001ebd60 con 0x7f83000c5770 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.195+0000 7f82ff7fe640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83001fed40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00b3f0 tx=0x7f82ec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.195+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec015020 con 0x7f83000c5770 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.195+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82ec011920 con 0x7f83000c5770 2024-09-17T05:34:40.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.196+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec011d50 con 0x7f83000c5770 2024-09-17T05:34:40.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.196+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83001ee200 con 0x7f83000c5770 2024-09-17T05:34:40.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.196+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83001ee4a0 con 0x7f83000c5770 2024-09-17T05:34:40.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.199+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f82ec018020 con 0x7f83000c5770 2024-09-17T05:34:40.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.200+0000 7f82fcff9640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f82e4078c10 0x7f82e407b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:40.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.200+0000 7f82fffff640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f82e4078c10 0x7f82e407b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:40.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.201+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e0005290 con 0x7f83000c5770 2024-09-17T05:34:40.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.205+0000 7f82fffff640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f82e4078c10 0x7f82e407b120 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f82f40095d0 tx=0x7f82f4006b30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:40.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.210+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f82ec05ed80 con 0x7f83000c5770 2024-09-17T05:34:40.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.217+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f82ec01f020 con 0x7f83000c5770 2024-09-17T05:34:40.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.659+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82e0005580 con 0x7f83000c5770 2024-09-17T05:34:40.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.660+0000 7f82fcff9640 1 -- 172.21.15.117:0/785040725 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f82ec063210 con 0x7f83000c5770 2024-09-17T05:34:40.668 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:40.668 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:40.668 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:40.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f82e4078c10 msgr2=0x7f82e407b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f82e4078c10 0x7f82e407b120 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f82f40095d0 tx=0x7f82f4006b30 comp rx=0 tx=0).stop 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 msgr2=0x7f83001fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83001fed40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00b3f0 tx=0x7f82ec00b8b0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 shutdown_connections 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f82e4078c10 0x7f82e407b120 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f830014dc50 0x7f83001ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83000c5770 0x7f83001fed40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 --2- 172.21.15.117:0/785040725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8300072b30 0x7f83001e08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:40.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 >> 172.21.15.117:0/785040725 conn(0x7f830006f0e0 msgr2=0x7f83000c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:40.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.669+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 shutdown_connections 2024-09-17T05:34:40.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:40.670+0000 7f8306769640 1 -- 172.21.15.117:0/785040725 wait complete. 2024-09-17T05:34:40.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:42.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:42.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.904+0000 7f57cc33a640 1 -- 172.21.15.117:0/3855824841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 msgr2=0x7f57c4075e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:42.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.904+0000 7f57cc33a640 1 --2- 172.21.15.117:0/3855824841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 0x7f57c4075e00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57c415bcf0 tx=0x7f57c002a580 comp rx=0 tx=0).stop 2024-09-17T05:34:42.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.905+0000 7f57cc33a640 1 -- 172.21.15.117:0/3855824841 shutdown_connections 2024-09-17T05:34:42.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.905+0000 7f57cc33a640 1 --2- 172.21.15.117:0/3855824841 >> v2:172.21.15.119:3300/0 conn(0x7f57c415abf0 0x7f57c415b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:42.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.905+0000 7f57cc33a640 1 --2- 172.21.15.117:0/3855824841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 0x7f57c4075e00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:42.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.905+0000 7f57cc33a640 1 --2- 172.21.15.117:0/3855824841 >> v2:172.21.15.117:3300/0 conn(0x7f57c40c6f50 0x7f57c4073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:42.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.905+0000 7f57cc33a640 1 -- 172.21.15.117:0/3855824841 >> 172.21.15.117:0/3855824841 conn(0x7f57c40c9a20 msgr2=0x7f57c40c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:42.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.908+0000 7f57cc33a640 1 -- 172.21.15.117:0/3855824841 shutdown_connections 2024-09-17T05:34:42.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.910+0000 7f57cc33a640 1 -- 172.21.15.117:0/3855824841 wait complete. 2024-09-17T05:34:42.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.912+0000 7f57cc33a640 1 Processor -- start 2024-09-17T05:34:42.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.914+0000 7f57cc33a640 1 -- start start 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.916+0000 7f57cc33a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c40c6f50 0x7f57c4152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.916+0000 7f57cc33a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 0x7f57c40c25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.916+0000 7f57cc33a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 0x7f57c416c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.916+0000 7f57cc33a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4157af0 con 0x7f57c415abf0 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.916+0000 7f57cc33a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f57c4157810 con 0x7f57c40c6f50 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.916+0000 7f57cc33a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f57c4157dd0 con 0x7f57c40c8390 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.917+0000 7f57c98d5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 0x7f57c40c25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.917+0000 7f57ca8d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 0x7f57c416c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.917+0000 7f57ca8d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 0x7f57c416c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52226/0 (socket says 172.21.15.117:52226) 2024-09-17T05:34:42.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.917+0000 7f57ca8d7640 1 -- 172.21.15.117:0/2146966579 learned_addr learned my addr 172.21.15.117:0/2146966579 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:42.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca0d6640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c40c6f50 0x7f57c4152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:42.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca8d7640 1 -- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 msgr2=0x7f57c40c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:42.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca8d7640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 0x7f57c40c25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:42.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca8d7640 1 -- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c40c6f50 msgr2=0x7f57c4152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:42.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca8d7640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c40c6f50 0x7f57c4152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:42.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca8d7640 1 -- 172.21.15.117:0/2146966579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c40cc4f0 con 0x7f57c415abf0 2024-09-17T05:34:42.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.918+0000 7f57ca8d7640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 0x7f57c416c630 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00b930 tx=0x7f57bc00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:42.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.919+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc019070 con 0x7f57c415abf0 2024-09-17T05:34:42.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.919+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57bc00cbd0 con 0x7f57c415abf0 2024-09-17T05:34:42.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.919+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc014440 con 0x7f57c415abf0 2024-09-17T05:34:42.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.921+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c40c2f40 con 0x7f57c415abf0 2024-09-17T05:34:42.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.921+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c40c20f0 con 0x7f57c415abf0 2024-09-17T05:34:42.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.922+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c40c3180 con 0x7f57c415abf0 2024-09-17T05:34:42.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.929+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f57bc01c070 con 0x7f57c415abf0 2024-09-17T05:34:42.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.930+0000 7f57baffd640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57a8078c60 0x7f57a807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:42.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.930+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f57bc09b550 con 0x7f57c415abf0 2024-09-17T05:34:42.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.932+0000 7f57ca0d6640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57a8078c60 0x7f57a807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:42.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.933+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f57c40c3180 con 0x7f57c415abf0 2024-09-17T05:34:42.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:42.934+0000 7f57ca0d6640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57a8078c60 0x7f57a807b170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005d80 tx=0x7f57b4005c70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:43.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.406+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57c40c94d0 con 0x7f57c415abf0 2024-09-17T05:34:43.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.409+0000 7f57baffd640 1 -- 172.21.15.117:0/2146966579 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f57bc062ff0 con 0x7f57c415abf0 2024-09-17T05:34:43.411 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:43.411 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:43.411 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:43.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.413+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57a8078c60 msgr2=0x7f57a807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:43.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.413+0000 7f57cc33a640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57a8078c60 0x7f57a807b170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005d80 tx=0x7f57b4005c70 comp rx=0 tx=0).stop 2024-09-17T05:34:43.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.413+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 msgr2=0x7f57c416c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:43.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.413+0000 7f57cc33a640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 0x7f57c416c630 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00b930 tx=0x7f57bc00bdf0 comp rx=0 tx=0).stop 2024-09-17T05:34:43.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 shutdown_connections 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f57a8078c60 0x7f57a807b170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c415abf0 0x7f57c416c630 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f57c40c8390 0x7f57c40c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 --2- 172.21.15.117:0/2146966579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57c40c6f50 0x7f57c4152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 >> 172.21.15.117:0/2146966579 conn(0x7f57c40c9a20 msgr2=0x7f57c4171330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 shutdown_connections 2024-09-17T05:34:43.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:43.416+0000 7f57cc33a640 1 -- 172.21.15.117:0/2146966579 wait complete. 2024-09-17T05:34:43.440 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-09-17T05:34:43.440 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:34:38:176058+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:34:37.169141+0000", "qdb_cluster": [ 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6837/1696536401", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 1696536401, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 1696536401, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5457, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:34:45.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:45.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.657+0000 7f773e7ce640 1 -- 172.21.15.117:0/292359646 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7724004800 msgr2=0x7f7724004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:45.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.657+0000 7f773e7ce640 1 --2- 172.21.15.117:0/292359646 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7724004800 0x7f7724004c10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7734006ed0 tx=0x7f7724005830 comp rx=0 tx=0).stop 2024-09-17T05:34:45.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.658+0000 7f773dfcd640 1 -- 172.21.15.117:0/292359646 reap_dead start 2024-09-17T05:34:45.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.658+0000 7f773e7ce640 1 -- 172.21.15.117:0/292359646 shutdown_connections 2024-09-17T05:34:45.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.658+0000 7f773e7ce640 1 -- 172.21.15.117:0/292359646 >> 172.21.15.117:0/292359646 conn(0x7f77380c8d10 msgr2=0x7f77380c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.663+0000 7f773e7ce640 1 -- 172.21.15.117:0/292359646 shutdown_connections 2024-09-17T05:34:45.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.664+0000 7f773e7ce640 1 -- 172.21.15.117:0/292359646 wait complete. 2024-09-17T05:34:45.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.667+0000 7f773e7ce640 1 Processor -- start 2024-09-17T05:34:45.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.670+0000 7f773e7ce640 1 -- start start 2024-09-17T05:34:45.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.672+0000 7f773e7ce640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f773814e410 0x7f7738152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:45.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.672+0000 7f773e7ce640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738150b00 0x7f7738152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:45.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.672+0000 7f773e7ce640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f773813f2b0 0x7f7738152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:45.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.672+0000 7f773e7ce640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7738074780 con 0x7f773813f2b0 2024-09-17T05:34:45.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.673+0000 7f773e7ce640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7738151860 con 0x7f7738150b00 2024-09-17T05:34:45.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.673+0000 7f773e7ce640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f773814ea60 con 0x7f773814e410 2024-09-17T05:34:45.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.673+0000 7f773d7cc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f773814e410 0x7f7738152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:45.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.673+0000 7f773d7cc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f773814e410 0x7f7738152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:50136/0 (socket says 172.21.15.117:50136) 2024-09-17T05:34:45.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.673+0000 7f773d7cc640 1 -- 172.21.15.117:0/1529454956 learned_addr learned my addr 172.21.15.117:0/1529454956 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:45.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.673+0000 7f773dfcd640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f773813f2b0 0x7f7738152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:45.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.674+0000 7f773dfcd640 1 -- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f773814e410 msgr2=0x7f7738152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:45.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.674+0000 7f773dfcd640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f773814e410 0x7f7738152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:45.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.674+0000 7f773dfcd640 1 -- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738150b00 msgr2=0x7f7738152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:45.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.674+0000 7f773dfcd640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738150b00 0x7f7738152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:45.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.674+0000 7f773dfcd640 1 -- 172.21.15.117:0/1529454956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7738142760 con 0x7f773813f2b0 2024-09-17T05:34:45.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.674+0000 7f773dfcd640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f773813f2b0 0x7f7738152cd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f77240039b0 tx=0x7f772800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:45.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.676+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728017070 con 0x7f773813f2b0 2024-09-17T05:34:45.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.676+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7728005a80 con 0x7f773813f2b0 2024-09-17T05:34:45.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.676+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280070d0 con 0x7f773813f2b0 2024-09-17T05:34:45.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.677+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7738151cc0 con 0x7f773813f2b0 2024-09-17T05:34:45.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.678+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7738153540 con 0x7f773813f2b0 2024-09-17T05:34:45.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.679+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7738141a10 con 0x7f773813f2b0 2024-09-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.692+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f772801a070 con 0x7f773813f2b0 2024-09-17T05:34:45.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.693+0000 7f772dffb640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f771c078c90 0x7f771c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:45.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.693+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f772809a340 con 0x7f773813f2b0 2024-09-17T05:34:45.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.693+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7728021020 con 0x7f773813f2b0 2024-09-17T05:34:45.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.698+0000 7f773d7cc640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f771c078c90 0x7f771c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:45.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:45.701+0000 7f773d7cc640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f771c078c90 0x7f771c07b1a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f77381425e0 tx=0x7f7730009640 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:46.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.155+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77380c45b0 con 0x7f773813f2b0 2024-09-17T05:34:46.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.159+0000 7f772dffb640 1 -- 172.21.15.117:0/1529454956 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f7728061de0 con 0x7f773813f2b0 2024-09-17T05:34:46.161 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:46.161 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:46.162 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:46.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f771c078c90 msgr2=0x7f771c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:46.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f771c078c90 0x7f771c07b1a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f77381425e0 tx=0x7f7730009640 comp rx=0 tx=0).stop 2024-09-17T05:34:46.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f773813f2b0 msgr2=0x7f7738152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:46.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f773813f2b0 0x7f7738152cd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f77240039b0 tx=0x7f772800c760 comp rx=0 tx=0).stop 2024-09-17T05:34:46.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 shutdown_connections 2024-09-17T05:34:46.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f771c078c90 0x7f771c07b1a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:46.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f773813f2b0 0x7f7738152cd0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:46.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738150b00 0x7f7738152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:46.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 --2- 172.21.15.117:0/1529454956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f773814e410 0x7f7738152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:46.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 >> 172.21.15.117:0/1529454956 conn(0x7f77380c8d10 msgr2=0x7f77380c7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:46.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 shutdown_connections 2024-09-17T05:34:46.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:46.164+0000 7f773e7ce640 1 -- 172.21.15.117:0/1529454956 wait complete. 2024-09-17T05:34:46.186 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:48.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:48.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/2714641358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f415bd10 msgr2=0x7fe9f415f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:48.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/2714641358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f415bd10 0x7fe9f415f120 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8005fd0 tx=0x7fe9e802a680 comp rx=0 tx=0).stop 2024-09-17T05:34:48.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/2714641358 shutdown_connections 2024-09-17T05:34:48.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/2714641358 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9f40c92c0 0x7fe9f40c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/2714641358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f415bd10 0x7fe9f415f120 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/2714641358 >> 172.21.15.117:0/2714641358 conn(0x7fe9f406d190 msgr2=0x7fe9f406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:48.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.384+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/2714641358 shutdown_connections 2024-09-17T05:34:48.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.385+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/2714641358 wait complete. 2024-09-17T05:34:48.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.388+0000 7fe9f8ef9640 1 Processor -- start 2024-09-17T05:34:48.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.388+0000 7fe9f8ef9640 1 -- start start 2024-09-17T05:34:48.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.390+0000 7fe9f8ef9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 0x7fe9f40cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:48.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.390+0000 7fe9f259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 0x7fe9f40cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:48.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.390+0000 7fe9f259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 0x7fe9f40cb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39766/0 (socket says 172.21.15.117:39766) 2024-09-17T05:34:48.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.390+0000 7fe9f259c640 1 -- 172.21.15.117:0/1479050545 learned_addr learned my addr 172.21.15.117:0/1479050545 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:48.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.390+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f40c92c0 0x7fe9f41e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:48.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.390+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9f415bd10 0x7fe9f414ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:48.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f415c120 con 0x7fe9f4070890 2024-09-17T05:34:48.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f4152090 con 0x7fe9f40c92c0 2024-09-17T05:34:48.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f4072e10 con 0x7fe9f415bd10 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f259c640 1 -- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9f415bd10 msgr2=0x7fe9f414ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f259c640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9f415bd10 0x7fe9f414ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f259c640 1 -- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f40c92c0 msgr2=0x7fe9f41e5e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f259c640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f40c92c0 0x7fe9f41e5e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.391+0000 7fe9f259c640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f41e58d0 con 0x7fe9f4070890 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.392+0000 7fe9f259c640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 0x7fe9f40cb8d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8034040 tx=0x7fe9e8003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.392+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e803e030 con 0x7fe9f4070890 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.393+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f41e6380 con 0x7fe9f4070890 2024-09-17T05:34:48.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.393+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9e80047a0 con 0x7fe9f4070890 2024-09-17T05:34:48.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.393+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8035040 con 0x7fe9f4070890 2024-09-17T05:34:48.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.393+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f41e5030 con 0x7fe9f4070890 2024-09-17T05:34:48.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.395+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f4155850 con 0x7fe9f4070890 2024-09-17T05:34:48.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.396+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe9e8041030 con 0x7fe9f4070890 2024-09-17T05:34:48.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.397+0000 7fe9e2ffd640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe9d4078bc0 0x7fe9d407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:48.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.397+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fe9e80bbb90 con 0x7fe9f4070890 2024-09-17T05:34:48.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.398+0000 7fe9f1d9b640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe9d4078bc0 0x7fe9d407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:48.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.401+0000 7fe9f1d9b640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe9d4078bc0 0x7fe9d407b0d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4155820 tx=0x7fe9e4006b80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:48.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.404+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe9e8083ba0 con 0x7fe9f4070890 2024-09-17T05:34:48.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.851+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9f40c6850 con 0x7fe9f4070890 2024-09-17T05:34:48.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.853+0000 7fe9e2ffd640 1 -- 172.21.15.117:0/1479050545 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7fe9e80834a0 con 0x7fe9f4070890 2024-09-17T05:34:48.857 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:48.857 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:48.858 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:48.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.858+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe9d4078bc0 msgr2=0x7fe9d407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:48.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.858+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe9d4078bc0 0x7fe9d407b0d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4155820 tx=0x7fe9e4006b80 comp rx=0 tx=0).stop 2024-09-17T05:34:48.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.858+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 msgr2=0x7fe9f40cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:48.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.858+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 0x7fe9f40cb8d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8034040 tx=0x7fe9e8003c60 comp rx=0 tx=0).stop 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 shutdown_connections 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe9d4078bc0 0x7fe9d407b0d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9f415bd10 0x7fe9f414ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f40c92c0 0x7fe9f41e5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 --2- 172.21.15.117:0/1479050545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f4070890 0x7fe9f40cb8d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 >> 172.21.15.117:0/1479050545 conn(0x7fe9f406d190 msgr2=0x7fe9f414f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 shutdown_connections 2024-09-17T05:34:48.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:48.859+0000 7fe9f8ef9640 1 -- 172.21.15.117:0/1479050545 wait complete. 2024-09-17T05:34:48.881 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:50.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:51.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.054+0000 7fa177ade640 1 -- 172.21.15.117:0/4273903980 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170072e10 msgr2=0x7fa17015ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:51.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.054+0000 7fa177ade640 1 --2- 172.21.15.117:0/4273903980 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170072e10 0x7fa17015ae60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa170071d40 tx=0x7fa16402a6f0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.055+0000 7fa177ade640 1 -- 172.21.15.117:0/4273903980 shutdown_connections 2024-09-17T05:34:51.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.055+0000 7fa177ade640 1 --2- 172.21.15.117:0/4273903980 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170072e10 0x7fa17015ae60 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.055+0000 7fa177ade640 1 --2- 172.21.15.117:0/4273903980 >> v2:172.21.15.119:3300/0 conn(0x7fa170157ea0 0x7fa1701517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.055+0000 7fa177ade640 1 -- 172.21.15.117:0/4273903980 >> 172.21.15.117:0/4273903980 conn(0x7fa17006d190 msgr2=0x7fa17006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:51.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.057+0000 7fa177ade640 1 -- 172.21.15.117:0/4273903980 shutdown_connections 2024-09-17T05:34:51.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.059+0000 7fa177ade640 1 -- 172.21.15.117:0/4273903980 wait complete. 2024-09-17T05:34:51.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.062+0000 7fa177ade640 1 Processor -- start 2024-09-17T05:34:51.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.062+0000 7fa177ade640 1 -- start start 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa177ade640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa170072e10 0x7fa17014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa177ade640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1700c6ca0 0x7fa1700cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa177ade640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 0x7fa170151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa177ade640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa1700c3820 con 0x7fa170157ea0 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa177ade640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa1700747f0 con 0x7fa170072e10 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa177ade640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa1700c67f0 con 0x7fa1700c6ca0 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa17607b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 0x7fa170151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa17607b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 0x7fa170151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39782/0 (socket says 172.21.15.117:39782) 2024-09-17T05:34:51.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.066+0000 7fa17607b640 1 -- 172.21.15.117:0/1179607595 learned_addr learned my addr 172.21.15.117:0/1179607595 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:51.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.067+0000 7fa17607b640 1 -- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1700c6ca0 msgr2=0x7fa1700cabe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.067+0000 7fa17607b640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1700c6ca0 0x7fa1700cabe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.067+0000 7fa17607b640 1 -- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa170072e10 msgr2=0x7fa17014d320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.067+0000 7fa17607b640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa170072e10 0x7fa17014d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.067+0000 7fa17607b640 1 -- 172.21.15.117:0/1179607595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17013bbc0 con 0x7fa170157ea0 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.067+0000 7fa17607b640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 0x7fa170151a00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c3c0 tx=0x7fa16000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.068+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160017070 con 0x7fa170157ea0 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.068+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1700cd020 con 0x7fa170157ea0 2024-09-17T05:34:51.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.068+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1600053c0 con 0x7fa170157ea0 2024-09-17T05:34:51.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.068+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160017070 con 0x7fa170157ea0 2024-09-17T05:34:51.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.068+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1700cba90 con 0x7fa170157ea0 2024-09-17T05:34:51.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.070+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa170156010 con 0x7fa170157ea0 2024-09-17T05:34:51.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.072+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa16001a070 con 0x7fa170157ea0 2024-09-17T05:34:51.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.073+0000 7fa16e7fc640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa15c078df0 0x7fa15c07b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:51.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.073+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fa1600981c0 con 0x7fa170157ea0 2024-09-17T05:34:51.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.074+0000 7fa17587a640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa15c078df0 0x7fa15c07b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:51.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.075+0000 7fa17587a640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa15c078df0 0x7fa15c07b300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa16800abb0 tx=0x7fa168009250 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:51.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.084+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa170156010 con 0x7fa170157ea0 2024-09-17T05:34:51.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.502+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1701517f0 con 0x7fa170157ea0 2024-09-17T05:34:51.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.504+0000 7fa16e7fc640 1 -- 172.21.15.117:0/1179607595 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7fa170156010 con 0x7fa170157ea0 2024-09-17T05:34:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:51.508 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:51.508 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:51.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.511+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa15c078df0 msgr2=0x7fa15c07b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:51.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.511+0000 7fa177ade640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa15c078df0 0x7fa15c07b300 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa16800abb0 tx=0x7fa168009250 comp rx=0 tx=0).stop 2024-09-17T05:34:51.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.511+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 msgr2=0x7fa170151a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:51.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.511+0000 7fa177ade640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 0x7fa170151a00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c3c0 tx=0x7fa16000c880 comp rx=0 tx=0).stop 2024-09-17T05:34:51.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.512+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 shutdown_connections 2024-09-17T05:34:51.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.512+0000 7fa177ade640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa15c078df0 0x7fa15c07b300 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.512+0000 7fa177ade640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa170157ea0 0x7fa170151a00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.512+0000 7fa177ade640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1700c6ca0 0x7fa1700cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.512+0000 7fa177ade640 1 --2- 172.21.15.117:0/1179607595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa170072e10 0x7fa17014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:51.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.512+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 >> 172.21.15.117:0/1179607595 conn(0x7fa17006d190 msgr2=0x7fa1700c5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:51.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.513+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 shutdown_connections 2024-09-17T05:34:51.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:51.513+0000 7fa177ade640 1 -- 172.21.15.117:0/1179607595 wait complete. 2024-09-17T05:34:51.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:53.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:53.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.738+0000 7f005294a640 1 -- 172.21.15.117:0/223983607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c9e60 msgr2=0x7f004c0c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:53.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.738+0000 7f005294a640 1 --2- 172.21.15.117:0/223983607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c9e60 0x7f004c0c3540 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0044005fd0 tx=0x7f004402a9a0 comp rx=0 tx=0).stop 2024-09-17T05:34:53.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.738+0000 7f005294a640 1 -- 172.21.15.117:0/223983607 shutdown_connections 2024-09-17T05:34:53.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.738+0000 7f005294a640 1 --2- 172.21.15.117:0/223983607 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:53.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.738+0000 7f005294a640 1 --2- 172.21.15.117:0/223983607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c9e60 0x7f004c0c3540 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0044005fd0 tx=0x7f004402a9a0 comp rx=0 tx=0).stop 2024-09-17T05:34:53.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.738+0000 7f005294a640 1 -- 172.21.15.117:0/223983607 >> 172.21.15.117:0/223983607 conn(0x7f004c06cda0 msgr2=0x7f004c06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:53.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.740+0000 7f005294a640 1 -- 172.21.15.117:0/223983607 shutdown_connections 2024-09-17T05:34:53.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.740+0000 7f005294a640 1 -- 172.21.15.117:0/223983607 wait complete. 2024-09-17T05:34:53.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.743+0000 7f005294a640 1 Processor -- start 2024-09-17T05:34:53.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.743+0000 7f005294a640 1 -- start start 2024-09-17T05:34:53.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.745+0000 7f005294a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c1f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:53.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f005294a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f004c14a4a0 0x7f004c1e89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:53.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f005294a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c6650 0x7f004c1482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:53.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f005294a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f004c0c3a80 con 0x7f004c14a4a0 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f005294a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f004c159540 con 0x7f004c0c6650 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f005294a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f004c0cad80 con 0x7f004c1530e0 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f004bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c1f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f004bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c1f9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:50178/0 (socket says 172.21.15.117:50178) 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.746+0000 7f004bfff640 1 -- 172.21.15.117:0/3818301905 learned_addr learned my addr 172.21.15.117:0/3818301905 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f0050ee7640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c6650 0x7f004c1482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:53.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004b7fe640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f004c14a4a0 0x7f004c1e89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:53.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004b7fe640 1 -- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 msgr2=0x7f004c1f9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004b7fe640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c1f9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004b7fe640 1 -- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c6650 msgr2=0x7f004c1482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004b7fe640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c6650 0x7f004c1482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004b7fe640 1 -- 172.21.15.117:0/3818301905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c1f9750 con 0x7f004c14a4a0 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f0050ee7640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c6650 0x7f004c1482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.747+0000 7f004bfff640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c1f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.748+0000 7f004b7fe640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f004c14a4a0 0x7f004c1e89c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0044032000 tx=0x7f0044003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.749+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044047070 con 0x7f004c14a4a0 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.749+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004c158ad0 con 0x7f004c14a4a0 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.749+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f004c0c7350 con 0x7f004c14a4a0 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.749+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f004402fb10 con 0x7f004c14a4a0 2024-09-17T05:34:53.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.749+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044047070 con 0x7f004c14a4a0 2024-09-17T05:34:53.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.752+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f002c005290 con 0x7f004c14a4a0 2024-09-17T05:34:53.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.753+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0044036070 con 0x7f004c14a4a0 2024-09-17T05:34:53.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.753+0000 7f0048ff9640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0038078dc0 0x7f003807b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:53.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.754+0000 7f004bfff640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0038078dc0 0x7f003807b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:53.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.754+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f004402fcc0 con 0x7f004c14a4a0 2024-09-17T05:34:53.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.754+0000 7f004bfff640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0038078dc0 0x7f003807b2d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0040006fd0 tx=0x7f0040008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:53.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:53.764+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0044087c60 con 0x7f004c14a4a0 2024-09-17T05:34:54.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.205+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f002c005850 con 0x7f004c14a4a0 2024-09-17T05:34:54.207 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:54.207 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:54.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.206+0000 7f0048ff9640 1 -- 172.21.15.117:0/3818301905 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f004408cb10 con 0x7f004c14a4a0 2024-09-17T05:34:54.208 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0038078dc0 msgr2=0x7f003807b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0038078dc0 0x7f003807b2d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0040006fd0 tx=0x7f0040008040 comp rx=0 tx=0).stop 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f004c14a4a0 msgr2=0x7f004c1e89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f004c14a4a0 0x7f004c1e89c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0044032000 tx=0x7f0044003f10 comp rx=0 tx=0).stop 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 shutdown_connections 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0038078dc0 0x7f003807b2d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f004c0c6650 0x7f004c1482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f004c14a4a0 0x7f004c1e89c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 --2- 172.21.15.117:0/3818301905 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f004c1530e0 0x7f004c1f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:54.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.210+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 >> 172.21.15.117:0/3818301905 conn(0x7f004c06cda0 msgr2=0x7f004c0c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:54.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.211+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 shutdown_connections 2024-09-17T05:34:54.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:54.211+0000 7f005294a640 1 -- 172.21.15.117:0/3818301905 wait complete. 2024-09-17T05:34:54.228 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:34:56.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.423+0000 7fcacc3da640 1 -- 172.21.15.117:0/3431381866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 msgr2=0x7fcac4070fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.423+0000 7fcacc3da640 1 --2- 172.21.15.117:0/3431381866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac4070fa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009830 tx=0x7fcac002a5b0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.424+0000 7fcacc3da640 1 -- 172.21.15.117:0/3431381866 shutdown_connections 2024-09-17T05:34:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.424+0000 7fcacc3da640 1 --2- 172.21.15.117:0/3431381866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac4070fa0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.424+0000 7fcacc3da640 1 --2- 172.21.15.117:0/3431381866 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac415bd10 0x7fcac415f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.424+0000 7fcacc3da640 1 -- 172.21.15.117:0/3431381866 >> 172.21.15.117:0/3431381866 conn(0x7fcac406d190 msgr2=0x7fcac406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:56.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.424+0000 7fcacc3da640 1 -- 172.21.15.117:0/3431381866 shutdown_connections 2024-09-17T05:34:56.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.424+0000 7fcacc3da640 1 -- 172.21.15.117:0/3431381866 wait complete. 2024-09-17T05:34:56.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.427+0000 7fcacc3da640 1 Processor -- start 2024-09-17T05:34:56.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.427+0000 7fcacc3da640 1 -- start start 2024-09-17T05:34:56.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.429+0000 7fcacc3da640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac40c92c0 0x7fcac40c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcacc3da640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac41f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcaca176640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac40c92c0 0x7fcac40c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcaca176640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac40c92c0 0x7fcac40c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:46568/0 (socket says 172.21.15.117:46568) 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcacc3da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcac415bd10 0x7fcac415d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcac9975640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac41f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcac9975640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac41f57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52404/0 (socket says 172.21.15.117:52404) 2024-09-17T05:34:56.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcaca176640 1 -- 172.21.15.117:0/3029461725 learned_addr learned my addr 172.21.15.117:0/3029461725 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcacc3da640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcac4152090 con 0x7fcac415bd10 2024-09-17T05:34:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcacc3da640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcac40c9ae0 con 0x7fcac40c96d0 2024-09-17T05:34:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcacc3da640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fcac415c120 con 0x7fcac40c92c0 2024-09-17T05:34:56.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.430+0000 7fcaca977640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcac415bd10 0x7fcac415d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.431+0000 7fcaca977640 1 -- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac40c92c0 msgr2=0x7fcac40c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.431+0000 7fcaca977640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac40c92c0 0x7fcac40c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.431+0000 7fcaca977640 1 -- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 msgr2=0x7fcac41f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.431+0000 7fcaca977640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac41f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.431+0000 7fcaca977640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac41f4d10 con 0x7fcac415bd10 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.432+0000 7fcaca977640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcac415bd10 0x7fcac415d960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcac0031000 tx=0x7fcac0033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.432+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0031b50 con 0x7fcac415bd10 2024-09-17T05:34:56.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.432+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcac0031de0 con 0x7fcac415bd10 2024-09-17T05:34:56.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.432+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0005260 con 0x7fcac415bd10 2024-09-17T05:34:56.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.432+0000 7fcacc3da640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac41e6e90 con 0x7fcac415bd10 2024-09-17T05:34:56.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.432+0000 7fcacc3da640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac41f4fb0 con 0x7fcac415bd10 2024-09-17T05:34:56.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.434+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac4150a70 con 0x7fcac415bd10 2024-09-17T05:34:56.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.436+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcac0005400 con 0x7fcac415bd10 2024-09-17T05:34:56.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.437+0000 7fcabaffd640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcaa8078c60 0x7fcaa807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:56.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.437+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fcac00bc580 con 0x7fcac415bd10 2024-09-17T05:34:56.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.438+0000 7fcaca176640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcaa8078c60 0x7fcaa807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:56.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.440+0000 7fcaca176640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcaa8078c60 0x7fcaa807b170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009f30 tx=0x7fcabc0095c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:56.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.443+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcac4150a70 con 0x7fcac415bd10 2024-09-17T05:34:56.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.856+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcac4072e10 con 0x7fcac415bd10 2024-09-17T05:34:56.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.857+0000 7fcabaffd640 1 -- 172.21.15.117:0/3029461725 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7fcac003f030 con 0x7fcac415bd10 2024-09-17T05:34:56.858 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:56.858 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:56.858 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcaa8078c60 msgr2=0x7fcaa807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcaa8078c60 0x7fcaa807b170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009f30 tx=0x7fcabc0095c0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcac415bd10 msgr2=0x7fcac415d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcac415bd10 0x7fcac415d960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcac0031000 tx=0x7fcac0033ba0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 shutdown_connections 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcaa8078c60 0x7fcaa807b170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcac415bd10 0x7fcac415d960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcac40c96d0 0x7fcac41f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 --2- 172.21.15.117:0/3029461725 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcac40c92c0 0x7fcac40c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 >> 172.21.15.117:0/3029461725 conn(0x7fcac406d190 msgr2=0x7fcac41e46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:56.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 shutdown_connections 2024-09-17T05:34:56.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:56.860+0000 7fcab8ff9640 1 -- 172.21.15.117:0/3029461725 wait complete. 2024-09-17T05:34:56.876 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2024-09-17T05:34:56.876 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:34:38:176058+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_4440": { "addr": "172.21.15.119:6839/2500745218", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6838", "nonce": 2500745218, "type": "v2" }, { "addr": "172.21.15.119:6839", "nonce": 2500745218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4440, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5232": { "addr": "172.21.15.117:6835/323067873", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 323067873, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 323067873, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5232, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:34:37.169141+0000", "qdb_cluster": [ 4440 ], "qdb_leader": 4440, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 4440, "mds_1": 5232 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6837/1696536401", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 1696536401, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 1696536401, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 37, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5457, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:34:58.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:34:59.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.136+0000 7f3dedcfa640 1 -- 172.21.15.117:0/3149468639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd4003a80 msgr2=0x7f3dd4003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:59.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.136+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/3149468639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd4003a80 0x7f3dd4003ef0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00016c0 tx=0x7f3dd4005830 comp rx=0 tx=0).stop 2024-09-17T05:34:59.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.138+0000 7f3de7fff640 1 -- 172.21.15.117:0/3149468639 reap_dead start 2024-09-17T05:34:59.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.141+0000 7f3dedcfa640 1 -- 172.21.15.117:0/3149468639 shutdown_connections 2024-09-17T05:34:59.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.141+0000 7f3dedcfa640 1 -- 172.21.15.117:0/3149468639 >> 172.21.15.117:0/3149468639 conn(0x7f3de8074b00 msgr2=0x7f3de80d0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:59.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.144+0000 7f3dedcfa640 1 -- 172.21.15.117:0/3149468639 shutdown_connections 2024-09-17T05:34:59.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.145+0000 7f3dedcfa640 1 -- 172.21.15.117:0/3149468639 wait complete. 2024-09-17T05:34:59.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.147+0000 7f3dedcfa640 1 Processor -- start 2024-09-17T05:34:59.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.148+0000 7f3dedcfa640 1 -- start start 2024-09-17T05:34:59.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.151+0000 7f3dedcfa640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 0x7f3de80c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.151+0000 7f3dedcfa640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3de815e0e0 0x7f3de81e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3dedcfa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3de8152c80 0x7f3de8153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3de77fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 0x7f3de80c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3de77fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 0x7f3de80c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:46584/0 (socket says 172.21.15.117:46584) 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3de77fe640 1 -- 172.21.15.117:0/1693553072 learned_addr learned my addr 172.21.15.117:0/1693553072 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3de806e2f0 con 0x7f3de8152c80 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3de7fff640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3de8152c80 0x7f3de8153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3de806e720 con 0x7f3de815e0e0 2024-09-17T05:34:59.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.152+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f3de806e470 con 0x7f3de815cef0 2024-09-17T05:34:59.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.153+0000 7f3ddffff640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3de815e0e0 0x7f3de81e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:59.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.154+0000 7f3de77fe640 1 -- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3de815e0e0 msgr2=0x7f3de81e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:59.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.154+0000 7f3de77fe640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3de815e0e0 0x7f3de81e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:59.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.154+0000 7f3de77fe640 1 -- 172.21.15.117:0/1693553072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3de8152c80 msgr2=0x7f3de8153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:59.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.154+0000 7f3de77fe640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3de8152c80 0x7f3de8153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:59.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.154+0000 7f3de77fe640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de815d3c0 con 0x7f3de815cef0 2024-09-17T05:34:59.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.154+0000 7f3de77fe640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 0x7f3de80c6d70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00017b0 tx=0x7f3dd002d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:59.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.156+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00035f0 con 0x7f3de815cef0 2024-09-17T05:34:59.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.157+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd0003790 con 0x7f3de815cef0 2024-09-17T05:34:59.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.157+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0003be0 con 0x7f3de815cef0 2024-09-17T05:34:59.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.157+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de80c5840 con 0x7f3de815cef0 2024-09-17T05:34:59.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.158+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de80c5e00 con 0x7f3de815cef0 2024-09-17T05:34:59.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.159+0000 7f3de7fff640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3de8152c80 0x7f3de8153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:34:59.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.162+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3dd003b030 con 0x7f3de815cef0 2024-09-17T05:34:59.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.163+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de80b6840 con 0x7f3de815cef0 2024-09-17T05:34:59.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.163+0000 7f3de4ff9640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3dc8078c60 0x7f3dc807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:34:59.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.163+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f3dd00bac10 con 0x7f3de815cef0 2024-09-17T05:34:59.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.165+0000 7f3ddffff640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3dc8078c60 0x7f3dc807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:34:59.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.166+0000 7f3ddffff640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3dc8078c60 0x7f3dc807b170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8005fd0 tx=0x7f3dd8005850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:34:59.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.173+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3de80b6840 con 0x7f3de815cef0 2024-09-17T05:34:59.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.624+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3de80c8670 con 0x7f3de815cef0 2024-09-17T05:34:59.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.628+0000 7f3de4ff9640 1 -- 172.21.15.117:0/1693553072 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f3de80c8670 con 0x7f3de815cef0 2024-09-17T05:34:59.631 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:34:59.631 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:34:59.631 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:34:59.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3dc8078c60 msgr2=0x7f3dc807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:59.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3dc8078c60 0x7f3dc807b170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8005fd0 tx=0x7f3dd8005850 comp rx=0 tx=0).stop 2024-09-17T05:34:59.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 msgr2=0x7f3de80c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:34:59.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 0x7f3de80c6d70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00017b0 tx=0x7f3dd002d000 comp rx=0 tx=0).stop 2024-09-17T05:34:59.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 shutdown_connections 2024-09-17T05:34:59.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3dc8078c60 0x7f3dc807b170 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8005fd0 tx=0x7f3dd8005850 comp rx=0 tx=0).stop 2024-09-17T05:34:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3de8152c80 0x7f3de8153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3de815e0e0 0x7f3de81e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 --2- 172.21.15.117:0/1693553072 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3de815cef0 0x7f3de80c6d70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:34:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.634+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 >> 172.21.15.117:0/1693553072 conn(0x7f3de8074b00 msgr2=0x7f3de8154e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:34:59.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.636+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 shutdown_connections 2024-09-17T05:34:59.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:34:59.637+0000 7f3dedcfa640 1 -- 172.21.15.117:0/1693553072 wait complete. 2024-09-17T05:34:59.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:01.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.862+0000 7f92a94ef640 1 -- 172.21.15.117:0/945311416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c27a0 msgr2=0x7f92a40cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.862+0000 7f92a94ef640 1 --2- 172.21.15.117:0/945311416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c27a0 0x7f92a40cc700 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f92a40c26c0 tx=0x7f929c032f10 comp rx=0 tx=0).stop 2024-09-17T05:35:01.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.863+0000 7f92a94ef640 1 -- 172.21.15.117:0/945311416 shutdown_connections 2024-09-17T05:35:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.863+0000 7f92a94ef640 1 --2- 172.21.15.117:0/945311416 >> v2:172.21.15.119:3300/0 conn(0x7f92a40cac90 0x7f92a40cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.863+0000 7f92a94ef640 1 --2- 172.21.15.117:0/945311416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c27a0 0x7f92a40cc700 secure :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f92a40c26c0 tx=0x7f929c032f10 comp rx=0 tx=0).stop 2024-09-17T05:35:01.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.863+0000 7f92a94ef640 1 -- 172.21.15.117:0/945311416 >> 172.21.15.117:0/945311416 conn(0x7f92a406cda0 msgr2=0x7f92a406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:01.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.869+0000 7f92a94ef640 1 -- 172.21.15.117:0/945311416 shutdown_connections 2024-09-17T05:35:01.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.871+0000 7f92a94ef640 1 -- 172.21.15.117:0/945311416 wait complete. 2024-09-17T05:35:01.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.874+0000 7f92a94ef640 1 Processor -- start 2024-09-17T05:35:01.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.874+0000 7f92a94ef640 1 -- start start 2024-09-17T05:35:01.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a94ef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 0x7f92a41edaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:01.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a94ef640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92a4155030 0x7f92a4146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:01.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a94ef640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c6410 0x7f92a40c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a94ef640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f92a4155440 con 0x7f92a40c6410 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a94ef640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f92a40cc1e0 con 0x7f92a40cac90 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a94ef640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f92a40764e0 con 0x7f92a4155030 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 0x7f92a41edaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 0x7f92a41edaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52432/0 (socket says 172.21.15.117:52432) 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.876+0000 7f92a3fff640 1 -- 172.21.15.117:0/2755858202 learned_addr learned my addr 172.21.15.117:0/2755858202 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:01.879 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.878+0000 7f92a8cee640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c6410 0x7f92a40c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.878+0000 7f92a37fe640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92a4155030 0x7f92a4146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.878+0000 7f92a3fff640 1 -- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92a4155030 msgr2=0x7f92a4146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.878+0000 7f92a3fff640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92a4155030 0x7f92a4146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:01.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.878+0000 7f92a3fff640 1 -- 172.21.15.117:0/2755858202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c6410 msgr2=0x7f92a40c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:01.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.881+0000 7f92a3fff640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c6410 0x7f92a40c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:01.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.881+0000 7f92a3fff640 1 -- 172.21.15.117:0/2755858202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a41edfe0 con 0x7f92a40cac90 2024-09-17T05:35:01.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.881+0000 7f92a8cee640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c6410 0x7f92a40c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:35:01.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.882+0000 7f92a3fff640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 0x7f92a41edaa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f929c013f20 tx=0x7f929c003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:01.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.884+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c049020 con 0x7f92a40cac90 2024-09-17T05:35:01.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.884+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f929c004640 con 0x7f92a40cac90 2024-09-17T05:35:01.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.885+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c035330 con 0x7f92a40cac90 2024-09-17T05:35:01.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.885+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a4146b60 con 0x7f92a40cac90 2024-09-17T05:35:01.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.886+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a40c4400 con 0x7f92a40cac90 2024-09-17T05:35:01.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.887+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a40c8eb0 con 0x7f92a40cac90 2024-09-17T05:35:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.890+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f929c04c020 con 0x7f92a40cac90 2024-09-17T05:35:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.891+0000 7f92a0ff9640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9284078be0 0x7f928407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:01.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.891+0000 7f92a37fe640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9284078be0 0x7f928407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:01.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.892+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f929c088900 con 0x7f92a40cac90 2024-09-17T05:35:01.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.893+0000 7f92a37fe640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9284078be0 0x7f928407b0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:01.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:01.908+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f92a40c8eb0 con 0x7f92a40cac90 2024-09-17T05:35:02.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.335+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92a406e070 con 0x7f92a40cac90 2024-09-17T05:35:02.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.336+0000 7f92a0ff9640 1 -- 172.21.15.117:0/2755858202 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f92a406e070 con 0x7f92a40cac90 2024-09-17T05:35:02.340 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:02.340 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:02.340 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9284078be0 msgr2=0x7f928407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9284078be0 0x7f928407b0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294008040 comp rx=0 tx=0).stop 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 msgr2=0x7f92a41edaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 0x7f92a41edaa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f929c013f20 tx=0x7f929c003f10 comp rx=0 tx=0).stop 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 shutdown_connections 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9284078be0 0x7f928407b0f0 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294008040 comp rx=0 tx=0).stop 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92a40c6410 0x7f92a40c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92a4155030 0x7f92a4146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:02.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 --2- 172.21.15.117:0/2755858202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92a40cac90 0x7f92a41edaa0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:02.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 >> 172.21.15.117:0/2755858202 conn(0x7f92a406cda0 msgr2=0x7f92a4151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:02.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 shutdown_connections 2024-09-17T05:35:02.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:02.346+0000 7f92a94ef640 1 -- 172.21.15.117:0/2755858202 wait complete. 2024-09-17T05:35:02.372 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:04.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:04.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.579+0000 7faced139640 1 -- 172.21.15.117:0/2704980068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 msgr2=0x7face815b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:04.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.579+0000 7faced139640 1 --2- 172.21.15.117:0/2704980068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face815b270 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7face8071d40 tx=0x7face002eb80 comp rx=0 tx=0).stop 2024-09-17T05:35:04.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.581+0000 7faced139640 1 -- 172.21.15.117:0/2704980068 shutdown_connections 2024-09-17T05:35:04.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.581+0000 7faced139640 1 --2- 172.21.15.117:0/2704980068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face815b270 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:04.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.581+0000 7faced139640 1 --2- 172.21.15.117:0/2704980068 >> v2:172.21.15.119:3300/0 conn(0x7face8157ea0 0x7face81517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:04.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.581+0000 7faced139640 1 -- 172.21.15.117:0/2704980068 >> 172.21.15.117:0/2704980068 conn(0x7face806d190 msgr2=0x7face806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:04.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.582+0000 7faced139640 1 -- 172.21.15.117:0/2704980068 shutdown_connections 2024-09-17T05:35:04.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.583+0000 7faced139640 1 -- 172.21.15.117:0/2704980068 wait complete. 2024-09-17T05:35:04.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.586+0000 7faced139640 1 Processor -- start 2024-09-17T05:35:04.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.586+0000 7faced139640 1 -- start start 2024-09-17T05:35:04.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7faced139640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7face80c6ca0 0x7face81e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:04.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7faced139640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7face8157ea0 0x7face814ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:04.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7faced139640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face814bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:04.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7faced139640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7face815b7b0 con 0x7face815ae60 2024-09-17T05:35:04.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7faced139640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7face80747f0 con 0x7face8157ea0 2024-09-17T05:35:04.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7faced139640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7face80c3820 con 0x7face80c6ca0 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7face759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face814bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7face759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face814bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34450/0 (socket says 172.21.15.117:34450) 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.588+0000 7face759e640 1 -- 172.21.15.117:0/3770271051 learned_addr learned my addr 172.21.15.117:0/3770271051 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.589+0000 7face759e640 1 -- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7face80c6ca0 msgr2=0x7face81e7b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.589+0000 7face759e640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7face80c6ca0 0x7face81e7b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.589+0000 7face759e640 1 -- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7face8157ea0 msgr2=0x7face814ce70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.589+0000 7face759e640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7face8157ea0 0x7face814ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.589+0000 7face759e640 1 -- 172.21.15.117:0/3770271051 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face81e80d0 con 0x7face815ae60 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.589+0000 7face759e640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face814bb20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7face002f170 tx=0x7face000a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.590+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face004e070 con 0x7face815ae60 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.590+0000 7faced139640 1 -- 172.21.15.117:0/3770271051 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face8149e70 con 0x7face815ae60 2024-09-17T05:35:04.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.591+0000 7faced139640 1 -- 172.21.15.117:0/3770271051 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face81ed950 con 0x7face815ae60 2024-09-17T05:35:04.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.595+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7face0012340 con 0x7face815ae60 2024-09-17T05:35:04.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.595+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0004300 con 0x7face815ae60 2024-09-17T05:35:04.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.595+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7face000f040 con 0x7face815ae60 2024-09-17T05:35:04.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.596+0000 7facdffff640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7facd4078de0 0x7facd407b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:04.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.596+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7face00cd8f0 con 0x7face815ae60 2024-09-17T05:35:04.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.602+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faccc005290 con 0x7face815ae60 2024-09-17T05:35:04.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.603+0000 7face6d9d640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7facd4078de0 0x7facd407b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:04.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.606+0000 7face6d9d640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7facd4078de0 0x7facd407b2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7facd80095d0 tx=0x7facd8006b30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:04.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:04.612+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7face815a980 con 0x7face815ae60 2024-09-17T05:35:05.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.105+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faccc005580 con 0x7face815ae60 2024-09-17T05:35:05.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.107+0000 7facdffff640 1 -- 172.21.15.117:0/3770271051 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7face00124f0 con 0x7face815ae60 2024-09-17T05:35:05.107 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:05.107 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:05.108 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:35:05.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.110+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7facd4078de0 msgr2=0x7facd407b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:05.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.110+0000 7facdd59a640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7facd4078de0 0x7facd407b2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7facd80095d0 tx=0x7facd8006b30 comp rx=0 tx=0).stop 2024-09-17T05:35:05.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.110+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 msgr2=0x7face814bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:05.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.110+0000 7facdd59a640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face814bb20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7face002f170 tx=0x7face000a1f0 comp rx=0 tx=0).stop 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 shutdown_connections 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7facd4078de0 0x7facd407b2f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face815ae60 0x7face814bb20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7face8157ea0 0x7face814ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 --2- 172.21.15.117:0/3770271051 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7face80c6ca0 0x7face81e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 >> 172.21.15.117:0/3770271051 conn(0x7face806d190 msgr2=0x7face80c83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:05.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 shutdown_connections 2024-09-17T05:35:05.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:05.111+0000 7facdd59a640 1 -- 172.21.15.117:0/3770271051 wait complete. 2024-09-17T05:35:05.132 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:07.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.344+0000 7f85f8757640 1 -- 172.21.15.117:0/197097875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f0070890 msgr2=0x7f85f0070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.344+0000 7f85f8757640 1 --2- 172.21.15.117:0/197097875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f0070890 0x7f85f0070ca0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f85e8005fd0 tx=0x7f85e802a8c0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.344+0000 7f85f8757640 1 -- 172.21.15.117:0/197097875 shutdown_connections 2024-09-17T05:35:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.344+0000 7f85f8757640 1 --2- 172.21.15.117:0/197097875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f0070890 0x7f85f0070ca0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.344+0000 7f85f8757640 1 --2- 172.21.15.117:0/197097875 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f00c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.344+0000 7f85f8757640 1 -- 172.21.15.117:0/197097875 >> 172.21.15.117:0/197097875 conn(0x7f85f006d190 msgr2=0x7f85f006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:07.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.347+0000 7f85f8757640 1 -- 172.21.15.117:0/197097875 shutdown_connections 2024-09-17T05:35:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.349+0000 7f85f8757640 1 -- 172.21.15.117:0/197097875 wait complete. 2024-09-17T05:35:07.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.352+0000 7f85f8757640 1 Processor -- start 2024-09-17T05:35:07.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.354+0000 7f85f8757640 1 -- start start 2024-09-17T05:35:07.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f8757640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85f0070890 0x7f85f01f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:07.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f8757640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f01e1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:07.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f8757640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f01581e0 0x7f85f0147220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:07.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f8757640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f85f00c5620 con 0x7f85f0070890 2024-09-17T05:35:07.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f8757640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f85f00c54a0 con 0x7f85f01581e0 2024-09-17T05:35:07.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f8757640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f85f00c57a0 con 0x7f85f00c8990 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f5cf2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f01e1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f5cf2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f01e1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:33962/0 (socket says 172.21.15.117:33962) 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.356+0000 7f85f5cf2640 1 -- 172.21.15.117:0/1738816345 learned_addr learned my addr 172.21.15.117:0/1738816345 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.358+0000 7f85f5cf2640 1 -- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f01581e0 msgr2=0x7f85f0147220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.358+0000 7f85f5cf2640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f01581e0 0x7f85f0147220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.358+0000 7f85f5cf2640 1 -- 172.21.15.117:0/1738816345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85f0070890 msgr2=0x7f85f01f5260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.358+0000 7f85f5cf2640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85f0070890 0x7f85f01f5260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.358+0000 7f85f5cf2640 1 -- 172.21.15.117:0/1738816345 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f01f57a0 con 0x7f85f00c8990 2024-09-17T05:35:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.360+0000 7f85f5cf2640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f01e1ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f85e400d6b0 tx=0x7f85e400db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:07.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.360+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4017070 con 0x7f85f00c8990 2024-09-17T05:35:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.361+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f0155cc0 con 0x7f85f00c8990 2024-09-17T05:35:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.361+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f00bfa10 con 0x7f85f00c8990 2024-09-17T05:35:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.361+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85e4004310 con 0x7f85f00c8990 2024-09-17T05:35:07.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.361+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4009a80 con 0x7f85f00c8990 2024-09-17T05:35:07.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.364+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f85e4009c20 con 0x7f85f00c8990 2024-09-17T05:35:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.365+0000 7f85eeffd640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f85dc078b90 0x7f85dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.365+0000 7f85f64f3640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f85dc078b90 0x7f85dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:07.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.366+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f85e409b1f0 con 0x7f85f00c8990 2024-09-17T05:35:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.367+0000 7f85f64f3640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f85dc078b90 0x7f85dc07b0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f85e00014d0 tx=0x7f85e00078a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:07.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.367+0000 7f85ecff9640 1 -- 172.21.15.117:0/1738816345 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f00c5c20 con 0x7f85f00c8990 2024-09-17T05:35:07.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.377+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f85e40631d0 con 0x7f85f00c8990 2024-09-17T05:35:07.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.825+0000 7f85ecff9640 1 -- 172.21.15.117:0/1738816345 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85f00c4dd0 con 0x7f85f00c8990 2024-09-17T05:35:07.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.828+0000 7f85eeffd640 1 -- 172.21.15.117:0/1738816345 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6907 (secure 0 0 0) 0x7f85e4062ad0 con 0x7f85f00c8990 2024-09-17T05:35:07.829 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:07.829 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":37,"btime":"2024-09-17T05:34:38:176058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:34:37.169141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":4440,"mds_1":5232},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5232":{"gid":5232,"name":"c","rank":1,"incarnation":27,"state":"up:stopping","state_seq":23,"addr":"172.21.15.117:6835/323067873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":323067873},{"type":"v1","addr":"172.21.15.117:6835","nonce":323067873}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:07.830 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 37 2024-09-17T05:35:07.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f85dc078b90 msgr2=0x7f85dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:07.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f85dc078b90 0x7f85dc07b0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f85e00014d0 tx=0x7f85e00078a0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 msgr2=0x7f85f01e1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:07.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f01e1ef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f85e400d6b0 tx=0x7f85e400db70 comp rx=0 tx=0).stop 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 shutdown_connections 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f85dc078b90 0x7f85dc07b0a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f01581e0 0x7f85f0147220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f85f00c8990 0x7f85f01e1ef0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 --2- 172.21.15.117:0/1738816345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85f0070890 0x7f85f01f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.834+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 >> 172.21.15.117:0/1738816345 conn(0x7f85f006d190 msgr2=0x7f85f01516f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.835+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 shutdown_connections 2024-09-17T05:35:07.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:07.835+0000 7f85f8757640 1 -- 172.21.15.117:0/1738816345 wait complete. 2024-09-17T05:35:07.857 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:08.039 INFO:tasks.ceph.mds.c.smithi117.stdout:starting mds.c at 2024-09-17T05:35:09.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:10.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.072+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4263556909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f88003a00 msgr2=0x7f0f88003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:10.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.072+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4263556909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f88003a00 0x7f0f88003e50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c310 tx=0x7f0f88005830 comp rx=0 tx=0).stop 2024-09-17T05:35:10.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.074+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4263556909 shutdown_connections 2024-09-17T05:35:10.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.074+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4263556909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f88003a00 0x7f0f88003e50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.074+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4263556909 >> 172.21.15.117:0/4263556909 conn(0x7f0fa00c9a20 msgr2=0x7f0fa00c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:10.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.077+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4263556909 shutdown_connections 2024-09-17T05:35:10.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.077+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4263556909 wait complete. 2024-09-17T05:35:10.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.080+0000 7f0fa63e4640 1 Processor -- start 2024-09-17T05:35:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.084+0000 7f0fa63e4640 1 -- start start 2024-09-17T05:35:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0fa63e4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f88003a00 0x7f0fa01ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0fa63e4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 0x7f0fa0152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0fa63e4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0fa014d440 0x7f0fa014d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0fa63e4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa0158140 con 0x7f0fa0151f30 2024-09-17T05:35:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0fa63e4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa0074370 con 0x7f0f88003a00 2024-09-17T05:35:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0fa63e4640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0fa0157af0 con 0x7f0fa014d440 2024-09-17T05:35:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0f9f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 0x7f0fa0152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0f9f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 0x7f0fa0152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34506/0 (socket says 172.21.15.117:34506) 2024-09-17T05:35:10.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.087+0000 7f0f9f7fe640 1 -- 172.21.15.117:0/4078895388 learned_addr learned my addr 172.21.15.117:0/4078895388 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.088+0000 7f0f9f7fe640 1 -- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0fa014d440 msgr2=0x7f0fa014d850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:35:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.088+0000 7f0f9f7fe640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0fa014d440 0x7f0fa014d850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.088+0000 7f0f9f7fe640 1 -- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f88003a00 msgr2=0x7f0fa01ed1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:35:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.088+0000 7f0f9f7fe640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f88003a00 0x7f0fa01ed1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.089+0000 7f0f9f7fe640 1 -- 172.21.15.117:0/4078895388 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fa01ed6e0 con 0x7f0fa0151f30 2024-09-17T05:35:10.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.089+0000 7f0f9f7fe640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 0x7f0fa0152c60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0070d70 tx=0x7f0f9000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:10.095 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.094+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f90015020 con 0x7f0fa0151f30 2024-09-17T05:35:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.094+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fa0155c30 con 0x7f0fa0151f30 2024-09-17T05:35:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.094+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fa01ed8f0 con 0x7f0fa0151f30 2024-09-17T05:35:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.095+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f90011920 con 0x7f0fa0151f30 2024-09-17T05:35:10.096 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.096+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f90011d50 con 0x7f0fa0151f30 2024-09-17T05:35:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.099+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f80005290 con 0x7f0fa0151f30 2024-09-17T05:35:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.103+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0f90018020 con 0x7f0fa0151f30 2024-09-17T05:35:10.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.103+0000 7f0f9cff9640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0f8c078c90 0x7f0f8c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.104+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f0f9009afb0 con 0x7f0fa0151f30 2024-09-17T05:35:10.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.105+0000 7f0f9ffff640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0f8c078c90 0x7f0f8c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:10.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.107+0000 7f0f9ffff640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0f8c078c90 0x7f0f8c07b1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c2e0 tx=0x7f0f940019f0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:10.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.108+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0f9001f020 con 0x7f0fa0151f30 2024-09-17T05:35:10.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.543+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f80005580 con 0x7f0fa0151f30 2024-09-17T05:35:10.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.545+0000 7f0f9cff9640 1 -- 172.21.15.117:0/4078895388 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7f0f90063010 con 0x7f0fa0151f30 2024-09-17T05:35:10.547 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:10.549 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:10.549 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0f8c078c90 msgr2=0x7f0f8c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0f8c078c90 0x7f0f8c07b1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400c2e0 tx=0x7f0f940019f0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 msgr2=0x7f0fa0152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 0x7f0fa0152c60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0070d70 tx=0x7f0f9000b870 comp rx=0 tx=0).stop 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 shutdown_connections 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0f8c078c90 0x7f0f8c07b1a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0fa014d440 0x7f0fa014d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fa0151f30 0x7f0fa0152c60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 --2- 172.21.15.117:0/4078895388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f88003a00 0x7f0fa01ed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 >> 172.21.15.117:0/4078895388 conn(0x7f0fa00c9a20 msgr2=0x7f0fa0156e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:10.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.552+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 shutdown_connections 2024-09-17T05:35:10.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.553+0000 7f0fa63e4640 1 -- 172.21.15.117:0/4078895388 wait complete. 2024-09-17T05:35:10.579 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 1 alive and active, now stable! 2024-09-17T05:35:10.579 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-09-17T05:35:10.579 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-09-17T05:35:10.579 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:35:10.596 INFO:tasks.ceph.mds.f:Stopped 2024-09-17T05:35:10.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:10.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.816+0000 7f004f4b6640 1 -- 172.21.15.117:0/4130092167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0048157ea0 msgr2=0x7f00481517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:10.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.816+0000 7f004f4b6640 1 --2- 172.21.15.117:0/4130092167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0048157ea0 0x7f00481517f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f00480725e0 tx=0x7f003802b270 comp rx=0 tx=0).stop 2024-09-17T05:35:10.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.819+0000 7f004f4b6640 1 -- 172.21.15.117:0/4130092167 shutdown_connections 2024-09-17T05:35:10.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.819+0000 7f004f4b6640 1 --2- 172.21.15.117:0/4130092167 >> v2:172.21.15.119:3300/0 conn(0x7f00480c6ca0 0x7f00480c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.819+0000 7f004f4b6640 1 --2- 172.21.15.117:0/4130092167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0048157ea0 0x7f00481517f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.819 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.819+0000 7f004f4b6640 1 -- 172.21.15.117:0/4130092167 >> 172.21.15.117:0/4130092167 conn(0x7f004806d190 msgr2=0x7f004806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:10.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.825+0000 7f004f4b6640 1 -- 172.21.15.117:0/4130092167 shutdown_connections 2024-09-17T05:35:10.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.825+0000 7f004f4b6640 1 -- 172.21.15.117:0/4130092167 wait complete. 2024-09-17T05:35:10.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.828+0000 7f004f4b6640 1 Processor -- start 2024-09-17T05:35:10.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.829+0000 7f004f4b6640 1 -- start start 2024-09-17T05:35:10.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.831+0000 7f004f4b6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 0x7f00481585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.831+0000 7f004f4b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 0x7f004815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004d252640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 0x7f00481585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:10.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004d252640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 0x7f00481585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44046/0 (socket says 172.21.15.117:44046) 2024-09-17T05:35:10.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004ca51640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 0x7f004815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:10.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004ca51640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 0x7f004815a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34536/0 (socket says 172.21.15.117:34536) 2024-09-17T05:35:10.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004d252640 1 -- 172.21.15.117:0/1017808710 learned_addr learned my addr 172.21.15.117:0/1017808710 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:10.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0048157ea0 0x7f004814c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.832+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f00480c3820 con 0x7f00480c9950 2024-09-17T05:35:10.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.833+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f00480747f0 con 0x7f00480c6ca0 2024-09-17T05:35:10.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.833+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f00480c67f0 con 0x7f0048157ea0 2024-09-17T05:35:10.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.833+0000 7f004da53640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0048157ea0 0x7f004814c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:10.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.833+0000 7f004ca51640 1 -- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0048157ea0 msgr2=0x7f004814c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:10.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.833+0000 7f004ca51640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0048157ea0 0x7f004814c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.834+0000 7f004ca51640 1 -- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 msgr2=0x7f00481585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:10.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.834+0000 7f004ca51640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 0x7f00481585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:10.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.834+0000 7f004ca51640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0048159c20 con 0x7f00480c9950 2024-09-17T05:35:10.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.834+0000 7f004d252640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 0x7f00481585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:35:10.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.834+0000 7f004ca51640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 0x7f004815a3e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f004000bf90 tx=0x7f0040009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:10.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.835+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040013600 con 0x7f00480c9950 2024-09-17T05:35:10.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.835+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00400137a0 con 0x7f00480c9950 2024-09-17T05:35:10.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.835+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040013600 con 0x7f00480c9950 2024-09-17T05:35:10.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.835+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00481f22d0 con 0x7f00480c9950 2024-09-17T05:35:10.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.835+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00481f2590 con 0x7f00480c9950 2024-09-17T05:35:10.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.837+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00481f20e0 con 0x7f00480c9950 2024-09-17T05:35:10.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.842+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f004001c400 con 0x7f00480c9950 2024-09-17T05:35:10.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.843+0000 7f003dffb640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f002c078d90 0x7f002c07b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:10.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.843+0000 7f004d252640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f002c078d90 0x7f002c07b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:10.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.843+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f0040099af0 con 0x7f00480c9950 2024-09-17T05:35:10.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.848+0000 7f004d252640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f002c078d90 0x7f002c07b2a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0038007320 tx=0x7f003802d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:10.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:10.853+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0040063a10 con 0x7f00480c9950 2024-09-17T05:35:11.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.314+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00481f29d0 con 0x7f00480c9950 2024-09-17T05:35:11.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.315+0000 7f003dffb640 1 -- 172.21.15.117:0/1017808710 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7f0040063a10 con 0x7f00480c9950 2024-09-17T05:35:11.316 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:11.316 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:11.317 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:11.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f002c078d90 msgr2=0x7f002c07b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:11.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f002c078d90 0x7f002c07b2a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0038007320 tx=0x7f003802d040 comp rx=0 tx=0).stop 2024-09-17T05:35:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 msgr2=0x7f004815a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 0x7f004815a3e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f004000bf90 tx=0x7f0040009f10 comp rx=0 tx=0).stop 2024-09-17T05:35:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 shutdown_connections 2024-09-17T05:35:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f002c078d90 0x7f002c07b2a0 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0038007320 tx=0x7f003802d040 comp rx=0 tx=0).stop 2024-09-17T05:35:11.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0048157ea0 0x7f004814c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00480c9950 0x7f004815a3e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 --2- 172.21.15.117:0/1017808710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00480c6ca0 0x7f00481585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.320+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 >> 172.21.15.117:0/1017808710 conn(0x7f004806d190 msgr2=0x7f00480c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.321+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 shutdown_connections 2024-09-17T05:35:11.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:11.321+0000 7f004f4b6640 1 -- 172.21.15.117:0/1017808710 wait complete. 2024-09-17T05:35:11.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-09-17T05:35:13.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:13.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.556+0000 7f1653407640 1 -- 172.21.15.117:0/210442813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 msgr2=0x7f164c0b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:13.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.556+0000 7f1653407640 1 --2- 172.21.15.117:0/210442813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c0b2740 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1640009830 tx=0x7f164002a5b0 comp rx=0 tx=0).stop 2024-09-17T05:35:13.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.562+0000 7f1653407640 1 -- 172.21.15.117:0/210442813 shutdown_connections 2024-09-17T05:35:13.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.562+0000 7f1653407640 1 --2- 172.21.15.117:0/210442813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c0b2740 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:13.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.562+0000 7f1653407640 1 -- 172.21.15.117:0/210442813 >> 172.21.15.117:0/210442813 conn(0x7f164c0c9a20 msgr2=0x7f164c0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:13.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.563+0000 7f1653407640 1 -- 172.21.15.117:0/210442813 shutdown_connections 2024-09-17T05:35:13.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.563+0000 7f1653407640 1 -- 172.21.15.117:0/210442813 wait complete. 2024-09-17T05:35:13.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.566+0000 7f1653407640 1 Processor -- start 2024-09-17T05:35:13.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.566+0000 7f1653407640 1 -- start start 2024-09-17T05:35:13.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f1653407640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f164c073f60 0x7f164c159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:13.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f1653407640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f164c0c6f50 0x7f164c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:13.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f1653407640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:13.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f1653407640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f164c158140 con 0x7f164c0c6f50 2024-09-17T05:35:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f1653407640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f164c074370 con 0x7f164c0c8390 2024-09-17T05:35:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f1653407640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f164c157af0 con 0x7f164c073f60 2024-09-17T05:35:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.569+0000 7f16519a4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.570+0000 7f16519a4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c15aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44056/0 (socket says 172.21.15.117:44056) 2024-09-17T05:35:13.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.570+0000 7f16519a4640 1 -- 172.21.15.117:0/3790069858 learned_addr learned my addr 172.21.15.117:0/3790069858 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.570+0000 7f16509a2640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f164c0c6f50 0x7f164c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.571+0000 7f16509a2640 1 -- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f164c073f60 msgr2=0x7f164c159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.571+0000 7f16509a2640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f164c073f60 0x7f164c159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.571+0000 7f16509a2640 1 -- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 msgr2=0x7f164c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.571+0000 7f16509a2640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c15aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.571+0000 7f16509a2640 1 -- 172.21.15.117:0/3790069858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164c1598b0 con 0x7f164c0c6f50 2024-09-17T05:35:13.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.571+0000 7f16509a2640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f164c0c6f50 0x7f164c1480c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f164c070d70 tx=0x7f164800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.572+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648015d70 con 0x7f164c0c6f50 2024-09-17T05:35:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.572+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16480050d0 con 0x7f164c0c6f50 2024-09-17T05:35:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.572+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648007020 con 0x7f164c0c6f50 2024-09-17T05:35:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.573+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f164c15bf30 con 0x7f164c0c6f50 2024-09-17T05:35:13.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.573+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f164c15c4f0 con 0x7f164c0c6f50 2024-09-17T05:35:13.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.577+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f164c073c80 con 0x7f164c0c6f50 2024-09-17T05:35:13.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.590+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f16480071c0 con 0x7f164c0c6f50 2024-09-17T05:35:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.591+0000 7f163dffb640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1630078c60 0x7f163007b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.591+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f16480a2d90 con 0x7f164c0c6f50 2024-09-17T05:35:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.591+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f16480d3d90 con 0x7f164c0c6f50 2024-09-17T05:35:13.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.592+0000 7f16511a3640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1630078c60 0x7f163007b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:13.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:13.599+0000 7f16511a3640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1630078c60 0x7f163007b170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1644004a30 tx=0x7f1644001650 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:14.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.057+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f164c0771f0 con 0x7f164c0c6f50 2024-09-17T05:35:14.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.059+0000 7f163dffb640 1 -- 172.21.15.117:0/3790069858 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7f164c0771f0 con 0x7f164c0c6f50 2024-09-17T05:35:14.062 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:14.064 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:14.064 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:14.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1630078c60 msgr2=0x7f163007b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:14.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1630078c60 0x7f163007b170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1644004a30 tx=0x7f1644001650 comp rx=0 tx=0).stop 2024-09-17T05:35:14.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f164c0c6f50 msgr2=0x7f164c1480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:14.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f164c0c6f50 0x7f164c1480c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f164c070d70 tx=0x7f164800c760 comp rx=0 tx=0).stop 2024-09-17T05:35:14.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 shutdown_connections 2024-09-17T05:35:14.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1630078c60 0x7f163007b170 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:14.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f164c0c8390 0x7f164c15aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:14.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f164c0c6f50 0x7f164c1480c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:14.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 --2- 172.21.15.117:0/3790069858 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f164c073f60 0x7f164c159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:14.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 >> 172.21.15.117:0/3790069858 conn(0x7f164c0c9a20 msgr2=0x7f164c075e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:14.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 shutdown_connections 2024-09-17T05:35:14.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:14.069+0000 7f1653407640 1 -- 172.21.15.117:0/3790069858 wait complete. 2024-09-17T05:35:14.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-09-17T05:35:16.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:16.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.294+0000 7f8839ebe640 1 -- 172.21.15.117:0/2078157375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 msgr2=0x7f881c00ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:16.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.294+0000 7f8839ebe640 1 --2- 172.21.15.117:0/2078157375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f881c00ce50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f882c001f60 tx=0x7f881c003810 comp rx=0 tx=0).stop 2024-09-17T05:35:16.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.294+0000 7f8839ebe640 1 -- 172.21.15.117:0/2078157375 shutdown_connections 2024-09-17T05:35:16.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.294+0000 7f8839ebe640 1 --2- 172.21.15.117:0/2078157375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f881c00ce50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.294+0000 7f8839ebe640 1 -- 172.21.15.117:0/2078157375 >> 172.21.15.117:0/2078157375 conn(0x7f883406d190 msgr2=0x7f883406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:16.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.296+0000 7f8839ebe640 1 -- 172.21.15.117:0/2078157375 shutdown_connections 2024-09-17T05:35:16.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.298+0000 7f8839ebe640 1 -- 172.21.15.117:0/2078157375 wait complete. 2024-09-17T05:35:16.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.301+0000 7f8839ebe640 1 Processor -- start 2024-09-17T05:35:16.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.301+0000 7f8839ebe640 1 -- start start 2024-09-17T05:35:16.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f8839ebe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f88341e1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:16.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f8839ebe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f883414ad60 0x7f8834158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:16.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f8839ebe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88341593d0 0x7f8834147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:16.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f88337fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f88341e1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:16.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f88337fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f88341e1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:56690/0 (socket says 172.21.15.117:56690) 2024-09-17T05:35:16.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f8839ebe640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88340c3820 con 0x7f88341593d0 2024-09-17T05:35:16.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.305+0000 7f88337fe640 1 -- 172.21.15.117:0/3526261624 learned_addr learned my addr 172.21.15.117:0/3526261624 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:16.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f8833fff640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88341593d0 0x7f8834147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:16.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f8839ebe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f88340747f0 con 0x7f881c0066a0 2024-09-17T05:35:16.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f8839ebe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f88340c67f0 con 0x7f883414ad60 2024-09-17T05:35:16.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f8832ffd640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f883414ad60 0x7f8834158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f88337fe640 1 -- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f883414ad60 msgr2=0x7f8834158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f88337fe640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f883414ad60 0x7f8834158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f88337fe640 1 -- 172.21.15.117:0/3526261624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88341593d0 msgr2=0x7f8834147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.306+0000 7f88337fe640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88341593d0 0x7f8834147d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.307+0000 7f88337fe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88341e1fb0 con 0x7f881c0066a0 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.307+0000 7f88337fe640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f88341e1a70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8828011b10 tx=0x7f882800f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.307+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882801e070 con 0x7f881c0066a0 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.307+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f882800fdd0 con 0x7f881c0066a0 2024-09-17T05:35:16.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.307+0000 7f8839ebe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88341e2730 con 0x7f881c0066a0 2024-09-17T05:35:16.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.308+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828019670 con 0x7f881c0066a0 2024-09-17T05:35:16.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.308+0000 7f8839ebe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88341e2f90 con 0x7f881c0066a0 2024-09-17T05:35:16.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.310+0000 7f8839ebe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8834075060 con 0x7f881c0066a0 2024-09-17T05:35:16.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.310+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8828021070 con 0x7f881c0066a0 2024-09-17T05:35:16.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.311+0000 7f881bfff640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8820078c00 0x7f882007b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:16.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.311+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f882809bc10 con 0x7f881c0066a0 2024-09-17T05:35:16.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.314+0000 7f8832ffd640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8820078c00 0x7f882007b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:16.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.316+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8834075060 con 0x7f881c0066a0 2024-09-17T05:35:16.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.319+0000 7f8832ffd640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8820078c00 0x7f882007b110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8834149310 tx=0x7f88240016a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:16.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.765+0000 7f8839ebe640 1 -- 172.21.15.117:0/3526261624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8834072e10 con 0x7f881c0066a0 2024-09-17T05:35:16.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.766+0000 7f881bfff640 1 -- 172.21.15.117:0/3526261624 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7f8828063620 con 0x7f881c0066a0 2024-09-17T05:35:16.768 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:16.768 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:16.769 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:16.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.770+0000 7f8819ffb640 1 -- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8820078c00 msgr2=0x7f882007b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:16.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.770+0000 7f8819ffb640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8820078c00 0x7f882007b110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8834149310 tx=0x7f88240016a0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.771+0000 7f8819ffb640 1 -- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 msgr2=0x7f88341e1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:16.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.771+0000 7f8819ffb640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f88341e1a70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8828011b10 tx=0x7f882800f3c0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 -- 172.21.15.117:0/3526261624 shutdown_connections 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8820078c00 0x7f882007b110 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88341593d0 0x7f8834147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f883414ad60 0x7f8834158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 --2- 172.21.15.117:0/3526261624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f881c0066a0 0x7f88341e1a70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 -- 172.21.15.117:0/3526261624 >> 172.21.15.117:0/3526261624 conn(0x7f883406d190 msgr2=0x7f88341e3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 -- 172.21.15.117:0/3526261624 shutdown_connections 2024-09-17T05:35:16.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:16.773+0000 7f8819ffb640 1 -- 172.21.15.117:0/3526261624 wait complete. 2024-09-17T05:35:16.797 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-09-17T05:35:18.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:18.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.969+0000 7fca1ffff640 1 -- 172.21.15.117:0/1820344846 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24031bf0 con 0x7fca280734e0 2024-09-17T05:35:18.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 -- 172.21.15.117:0/1820344846 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280734e0 msgr2=0x7fca2814e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:18.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 --2- 172.21.15.117:0/1820344846 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280734e0 0x7fca2814e880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fca24009830 tx=0x7fca2402a5a0 comp rx=0 tx=0).stop 2024-09-17T05:35:18.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 -- 172.21.15.117:0/1820344846 shutdown_connections 2024-09-17T05:35:18.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 --2- 172.21.15.117:0/1820344846 >> v2:172.21.15.119:3300/0 conn(0x7fca28072e10 0x7fca280c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:18.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 --2- 172.21.15.117:0/1820344846 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280734e0 0x7fca2814e880 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:18.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 -- 172.21.15.117:0/1820344846 >> 172.21.15.117:0/1820344846 conn(0x7fca2806d190 msgr2=0x7fca2806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:18.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 -- 172.21.15.117:0/1820344846 shutdown_connections 2024-09-17T05:35:18.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.971+0000 7fca2f62e640 1 -- 172.21.15.117:0/1820344846 wait complete. 2024-09-17T05:35:18.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.973+0000 7fca2f62e640 1 Processor -- start 2024-09-17T05:35:18.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.973+0000 7fca2f62e640 1 -- start start 2024-09-17T05:35:18.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.975+0000 7fca2f62e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca28072e10 0x7fca28145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:18.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.975+0000 7fca2f62e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca280734e0 0x7fca28154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2f62e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280c4230 0x7fca280ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2d3ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca28072e10 0x7fca28145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2d3ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca28072e10 0x7fca28145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40910/0 (socket says 172.21.15.117:40910) 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2d3ca640 1 -- 172.21.15.117:0/4038259268 learned_addr learned my addr 172.21.15.117:0/4038259268 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca28075100 con 0x7fca28072e10 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fca28074e20 con 0x7fca280734e0 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2dbcb640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280c4230 0x7fca280ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:18.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.976+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fca2814edc0 con 0x7fca280c4230 2024-09-17T05:35:18.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca2dbcb640 1 -- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca280734e0 msgr2=0x7fca28154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:18.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca2dbcb640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca280734e0 0x7fca28154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:18.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca2dbcb640 1 -- 172.21.15.117:0/4038259268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca28072e10 msgr2=0x7fca28145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:18.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca2dbcb640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca28072e10 0x7fca28145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:18.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca2dbcb640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca28155160 con 0x7fca280c4230 2024-09-17T05:35:18.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca2dbcb640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280c4230 0x7fca280ca200 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fca2000ca60 tx=0x7fca2000cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:18.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.977+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2001e070 con 0x7fca280c4230 2024-09-17T05:35:18.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.978+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca28154720 con 0x7fca280c4230 2024-09-17T05:35:18.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.978+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca280cc420 con 0x7fca280c4230 2024-09-17T05:35:18.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.979+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca20007e40 con 0x7fca280c4230 2024-09-17T05:35:18.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.979+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca200119b0 con 0x7fca280c4230 2024-09-17T05:35:18.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.980+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca08005290 con 0x7fca280c4230 2024-09-17T05:35:18.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.981+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fca20021070 con 0x7fca280c4230 2024-09-17T05:35:18.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.982+0000 7fca1dffb640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fca0c078b90 0x7fca0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:18.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.982+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7fca2009f580 con 0x7fca280c4230 2024-09-17T05:35:18.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.982+0000 7fca2d3ca640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fca0c078b90 0x7fca0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:18.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.983+0000 7fca2d3ca640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fca0c078b90 0x7fca0c07b0a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fca24005eb0 tx=0x7fca2403d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:18.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:18.991+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fca20066f10 con 0x7fca280c4230 2024-09-17T05:35:19.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.420+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca08005580 con 0x7fca280c4230 2024-09-17T05:35:19.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.422+0000 7fca1dffb640 1 -- 172.21.15.117:0/4038259268 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7fca2006b940 con 0x7fca280c4230 2024-09-17T05:35:19.424 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:19.424 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:19.424 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:19.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.426+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fca0c078b90 msgr2=0x7fca0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:19.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.426+0000 7fca2f62e640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fca0c078b90 0x7fca0c07b0a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fca24005eb0 tx=0x7fca2403d040 comp rx=0 tx=0).stop 2024-09-17T05:35:19.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.426+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280c4230 msgr2=0x7fca280ca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:19.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.426+0000 7fca2f62e640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280c4230 0x7fca280ca200 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fca2000ca60 tx=0x7fca2000cf20 comp rx=0 tx=0).stop 2024-09-17T05:35:19.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 shutdown_connections 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fca0c078b90 0x7fca0c07b0a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca280c4230 0x7fca280ca200 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca280734e0 0x7fca28154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 --2- 172.21.15.117:0/4038259268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca28072e10 0x7fca28145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 >> 172.21.15.117:0/4038259268 conn(0x7fca2806d190 msgr2=0x7fca280c7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 shutdown_connections 2024-09-17T05:35:19.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:19.427+0000 7fca2f62e640 1 -- 172.21.15.117:0/4038259268 wait complete. 2024-09-17T05:35:19.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-09-17T05:35:21.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:21.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.618+0000 7f52b1ffb640 1 -- 172.21.15.117:0/1540324219 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a40047a0 con 0x7f52b406f120 2024-09-17T05:35:21.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.619+0000 7f52b3fff640 1 -- 172.21.15.117:0/1540324219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b406f120 msgr2=0x7f52b4151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:21.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.619+0000 7f52b3fff640 1 --2- 172.21.15.117:0/1540324219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b406f120 0x7f52b4151c60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f52b40bffe0 tx=0x7f52a402a530 comp rx=0 tx=0).stop 2024-09-17T05:35:21.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.620+0000 7f52b3fff640 1 -- 172.21.15.117:0/1540324219 shutdown_connections 2024-09-17T05:35:21.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.620+0000 7f52b3fff640 1 --2- 172.21.15.117:0/1540324219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b406f120 0x7f52b4151c60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:21.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.620+0000 7f52b3fff640 1 -- 172.21.15.117:0/1540324219 >> 172.21.15.117:0/1540324219 conn(0x7f52b40c8d10 msgr2=0x7f52b40c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:21.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.620+0000 7f52b3fff640 1 -- 172.21.15.117:0/1540324219 shutdown_connections 2024-09-17T05:35:21.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.620+0000 7f52b3fff640 1 -- 172.21.15.117:0/1540324219 wait complete. 2024-09-17T05:35:21.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.623+0000 7f52b3fff640 1 Processor -- start 2024-09-17T05:35:21.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.629+0000 7f52b3fff640 1 -- start start 2024-09-17T05:35:21.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 0x7f52b41fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:21.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 0x7f52b4153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:21.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52b40746b0 0x7f52b4154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:21.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b3fff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f52b4073aa0 con 0x7f52b406f120 2024-09-17T05:35:21.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b3fff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f52b40737c0 con 0x7f52b40742a0 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b3fff640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f52b4073d80 con 0x7f52b40746b0 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 0x7f52b41fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 0x7f52b41fbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40916/0 (socket says 172.21.15.117:40916) 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b37fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52b40746b0 0x7f52b4154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b27fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 0x7f52b4153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.632+0000 7f52b27fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 0x7f52b4153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:56724/0 (socket says 172.21.15.117:56724) 2024-09-17T05:35:21.635 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.633+0000 7f52b27fc640 1 -- 172.21.15.117:0/582176979 learned_addr learned my addr 172.21.15.117:0/582176979 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:21.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.633+0000 7f52b27fc640 1 -- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52b40746b0 msgr2=0x7f52b4154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:21.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.633+0000 7f52b27fc640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52b40746b0 0x7f52b4154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:21.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.633+0000 7f52b27fc640 1 -- 172.21.15.117:0/582176979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 msgr2=0x7f52b41fbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:21.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.633+0000 7f52b27fc640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 0x7f52b41fbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:21.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.633+0000 7f52b27fc640 1 -- 172.21.15.117:0/582176979 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b414b2e0 con 0x7f52b40742a0 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.634+0000 7f52b37fe640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52b40746b0 0x7f52b4154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.634+0000 7f52b2ffd640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 0x7f52b41fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.634+0000 7f52b27fc640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 0x7f52b4153e30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f52a800c3c0 tx=0x7f52a800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.634+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a8011790 con 0x7f52b40742a0 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.634+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b4202010 con 0x7f52b40742a0 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.635+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b40c52d0 con 0x7f52b40742a0 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.635+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52a8017070 con 0x7f52b40742a0 2024-09-17T05:35:21.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.635+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a8011d20 con 0x7f52b40742a0 2024-09-17T05:35:21.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.636+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b415bbc0 con 0x7f52b40742a0 2024-09-17T05:35:21.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.638+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f52a801c460 con 0x7f52b40742a0 2024-09-17T05:35:21.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.638+0000 7f529bfff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f52a0078b90 0x7f52a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:21.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.638+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f52a809aed0 con 0x7f52b40742a0 2024-09-17T05:35:21.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.639+0000 7f52b2ffd640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f52a0078b90 0x7f52a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:21.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.641+0000 7f52b2ffd640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f52a0078b90 0x7f52a007b0a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f52a4033000 tx=0x7f52a4004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:21.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:21.647+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f52a8062860 con 0x7f52b40742a0 2024-09-17T05:35:22.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.028+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52b406a1d0 con 0x7f52b40742a0 2024-09-17T05:35:22.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.028+0000 7f529bfff640 1 -- 172.21.15.117:0/582176979 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7f52a8067290 con 0x7f52b40742a0 2024-09-17T05:35:22.030 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:22.030 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:22.031 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f52a0078b90 msgr2=0x7f52a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f52a0078b90 0x7f52a007b0a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f52a4033000 tx=0x7f52a4004730 comp rx=0 tx=0).stop 2024-09-17T05:35:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 msgr2=0x7f52b4153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 0x7f52b4153e30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f52a800c3c0 tx=0x7f52a800c880 comp rx=0 tx=0).stop 2024-09-17T05:35:22.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 shutdown_connections 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f52a0078b90 0x7f52a007b0a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52b40746b0 0x7f52b4154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52b40742a0 0x7f52b4153e30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.031+0000 7f52b3fff640 1 --2- 172.21.15.117:0/582176979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52b406f120 0x7f52b41fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.032+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 >> 172.21.15.117:0/582176979 conn(0x7f52b40c8d10 msgr2=0x7f52b4150f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.032+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 shutdown_connections 2024-09-17T05:35:22.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:22.032+0000 7f52b3fff640 1 -- 172.21.15.117:0/582176979 wait complete. 2024-09-17T05:35:22.049 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-09-17T05:35:24.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:24.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.276+0000 7f1b2365c640 1 -- 172.21.15.117:0/3714847255 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c7120 msgr2=0x7f1b1c0cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:24.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.276+0000 7f1b2365c640 1 --2- 172.21.15.117:0/3714847255 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c7120 0x7f1b1c0cbb00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c005fd0 tx=0x7f1b0c02a580 comp rx=0 tx=0).stop 2024-09-17T05:35:24.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.277+0000 7f1b2365c640 1 -- 172.21.15.117:0/3714847255 shutdown_connections 2024-09-17T05:35:24.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.277+0000 7f1b2365c640 1 --2- 172.21.15.117:0/3714847255 >> v2:172.21.15.119:3300/0 conn(0x7f1b1c0c5ad0 0x7f1b1c0c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.277+0000 7f1b2365c640 1 --2- 172.21.15.117:0/3714847255 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c7120 0x7f1b1c0cbb00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.277+0000 7f1b2365c640 1 -- 172.21.15.117:0/3714847255 >> 172.21.15.117:0/3714847255 conn(0x7f1b1c06e020 msgr2=0x7f1b1c06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:24.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.280+0000 7f1b2365c640 1 -- 172.21.15.117:0/3714847255 shutdown_connections 2024-09-17T05:35:24.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.280+0000 7f1b2365c640 1 -- 172.21.15.117:0/3714847255 wait complete. 2024-09-17T05:35:24.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.283+0000 7f1b2365c640 1 Processor -- start 2024-09-17T05:35:24.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.287+0000 7f1b2365c640 1 -- start start 2024-09-17T05:35:24.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.289+0000 7f1b2365c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c5ad0 0x7f1b1c1f6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:24.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.289+0000 7f1b2365c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 0x7f1b1c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:24.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.289+0000 7f1b213f8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c5ad0 0x7f1b1c1f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:24.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b213f8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c5ad0 0x7f1b1c1f6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:36890/0 (socket says 172.21.15.117:36890) 2024-09-17T05:35:24.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b20bf7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 0x7f1b1c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:24.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b20bf7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 0x7f1b1c14d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:35136/0 (socket says 172.21.15.117:35136) 2024-09-17T05:35:24.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b2365c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 0x7f1b1c15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:24.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b21bf9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 0x7f1b1c15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:24.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b21bf9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 0x7f1b1c15d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:48160/0 (socket says 172.21.15.117:48160) 2024-09-17T05:35:24.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b213f8640 1 -- 172.21.15.117:0/4288929594 learned_addr learned my addr 172.21.15.117:0/4288929594 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:24.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c15bbf0 con 0x7f1b1c15e870 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c15b910 con 0x7f1b1c0c7120 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.290+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1c0cc040 con 0x7f1b1c0c5ad0 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.291+0000 7f1b21bf9640 1 -- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c5ad0 msgr2=0x7f1b1c1f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.291+0000 7f1b21bf9640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c5ad0 0x7f1b1c1f6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.291+0000 7f1b21bf9640 1 -- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 msgr2=0x7f1b1c14d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.291+0000 7f1b21bf9640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 0x7f1b1c14d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.291+0000 7f1b21bf9640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1c1f6590 con 0x7f1b1c15e870 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.292+0000 7f1b21bf9640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 0x7f1b1c15d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c08ec20 tx=0x7f1b1800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.292+0000 7f1b20bf7640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 0x7f1b1c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.293+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18017070 con 0x7f1b1c15e870 2024-09-17T05:35:24.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.293+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b1c0d1520 con 0x7f1b1c15e870 2024-09-17T05:35:24.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.296+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b18005070 con 0x7f1b1c15e870 2024-09-17T05:35:24.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.297+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1800a640 con 0x7f1b1c15e870 2024-09-17T05:35:24.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.297+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1b1801a070 con 0x7f1b1c15e870 2024-09-17T05:35:24.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.297+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b1c0d0080 con 0x7f1b1c15e870 2024-09-17T05:35:24.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.301+0000 7f1b11ffb640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1b00078c10 0x7f1b0007b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:24.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.302+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6848+0+0 (secure 0 0 0) 0x7f1b1809a2c0 con 0x7f1b1c15e870 2024-09-17T05:35:24.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.302+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b1c0b6e50 con 0x7f1b1c15e870 2024-09-17T05:35:24.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.303+0000 7f1b213f8640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1b00078c10 0x7f1b0007b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:24.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.310+0000 7f1b213f8640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1b00078c10 0x7f1b0007b120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c000c00 tx=0x7f1b0c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:24.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.314+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1b1c0b6e50 con 0x7f1b1c15e870 2024-09-17T05:35:24.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.779+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b1c0c5ee0 con 0x7f1b1c15e870 2024-09-17T05:35:24.783 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.781+0000 7f1b11ffb640 1 -- 172.21.15.117:0/4288929594 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6895 (secure 0 0 0) 0x7f1b18062420 con 0x7f1b1c15e870 2024-09-17T05:35:24.783 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:24.784 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":39,"btime":"2024-09-17T05:35:08:937198+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5457,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:07.173991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4440},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_4440":{"gid":4440,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6839/2500745218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2500745218},{"type":"v1","addr":"172.21.15.119:6839","nonce":2500745218}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4440,"qdb_cluster":[4440]},"id":1}]} 2024-09-17T05:35:24.784 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 39 2024-09-17T05:35:24.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1b00078c10 msgr2=0x7f1b0007b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:24.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1b00078c10 0x7f1b0007b120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c000c00 tx=0x7f1b0c004730 comp rx=0 tx=0).stop 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 msgr2=0x7f1b1c15d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 0x7f1b1c15d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c08ec20 tx=0x7f1b1800cbe0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 shutdown_connections 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1b00078c10 0x7f1b0007b120 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b1c15e870 0x7f1b1c15d9c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b1c0c7120 0x7f1b1c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 --2- 172.21.15.117:0/4288929594 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1b1c0c5ad0 0x7f1b1c1f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:24.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.786+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 >> 172.21.15.117:0/4288929594 conn(0x7f1b1c06e020 msgr2=0x7f1b1c0c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:24.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.787+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 shutdown_connections 2024-09-17T05:35:24.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:24.787+0000 7f1b2365c640 1 -- 172.21.15.117:0/4288929594 wait complete. 2024-09-17T05:35:24.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-09-17T05:35:26.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:27.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.016+0000 7fc9c2a01640 1 -- 172.21.15.117:0/3651369714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b40d5450 msgr2=0x7fc9b40068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.016+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/3651369714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b40d5450 0x7fc9b40068e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4033b40 tx=0x7fc9ac02a600 comp rx=0 tx=0).stop 2024-09-17T05:35:27.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.017+0000 7fc9c2a01640 1 -- 172.21.15.117:0/3651369714 shutdown_connections 2024-09-17T05:35:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.017+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/3651369714 >> v2:172.21.15.119:3300/0 conn(0x7fc9b40db580 0x7fc9b40db990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.017+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/3651369714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b40d5450 0x7fc9b40068e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.017+0000 7fc9c2a01640 1 -- 172.21.15.117:0/3651369714 >> 172.21.15.117:0/3651369714 conn(0x7fc9b4050280 msgr2=0x7fc9b4050690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.018+0000 7fc9c2a01640 1 -- 172.21.15.117:0/3651369714 shutdown_connections 2024-09-17T05:35:27.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.019+0000 7fc9c2a01640 1 -- 172.21.15.117:0/3651369714 wait complete. 2024-09-17T05:35:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.023+0000 7fc9c2a01640 1 Processor -- start 2024-09-17T05:35:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.023+0000 7fc9c2a01640 1 -- start start 2024-09-17T05:35:27.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.025+0000 7fc9c2a01640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b4033290 0x7fc9b4049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.025+0000 7fc9c2a01640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9b40d5450 0x7fc9b40c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.025+0000 7fc9c2a01640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 0x7fc9b40d3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.025+0000 7fc9c2a01640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b404c640 con 0x7fc9b4033290 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.025+0000 7fc9c2a01640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b404c360 con 0x7fc9b40db580 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.025+0000 7fc9c2a01640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b40dbed0 con 0x7fc9b40d5450 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 0x7fc9b40d3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 0x7fc9b40d3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:35152/0 (socket says 172.21.15.117:35152) 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 -- 172.21.15.117:0/1335051543 learned_addr learned my addr 172.21.15.117:0/1335051543 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c19ff640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b4033290 0x7fc9b4049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 -- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9b40d5450 msgr2=0x7fc9b40c4980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:35:27.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9b40d5450 0x7fc9b40c4980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 -- 172.21.15.117:0/1335051543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b4033290 msgr2=0x7fc9b4049c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b4033290 0x7fc9b4049c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.026+0000 7fc9c2200640 1 -- 172.21.15.117:0/1335051543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b40c4ec0 con 0x7fc9b40db580 2024-09-17T05:35:27.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.028+0000 7fc9c2200640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 0x7fc9b40d3f30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc06bdf0 tx=0x7fc9bc06a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:27.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.029+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc06afb0 con 0x7fc9b40db580 2024-09-17T05:35:27.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.029+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9bc074a10 con 0x7fc9b40db580 2024-09-17T05:35:27.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.029+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc074dc0 con 0x7fc9b40db580 2024-09-17T05:35:27.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.031+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b40c5ca0 con 0x7fc9b40db580 2024-09-17T05:35:27.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.031+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b40c5f40 con 0x7fc9b40db580 2024-09-17T05:35:27.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.033+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4007e90 con 0x7fc9b40db580 2024-09-17T05:35:27.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.037+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9bc096460 con 0x7fc9b40db580 2024-09-17T05:35:27.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.038+0000 7fc9b27fc640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9a0078b90 0x7fc9a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.038+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fc9bc118300 con 0x7fc9b40db580 2024-09-17T05:35:27.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.040+0000 7fc9c19ff640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9a0078b90 0x7fc9a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.042+0000 7fc9c19ff640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9a0078b90 0x7fc9a007b0a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8000f60 tx=0x7fc9b8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:27.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.046+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9bc0e01e0 con 0x7fc9b40db580 2024-09-17T05:35:27.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.479+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9b417dc80 con 0x7fc9b40db580 2024-09-17T05:35:27.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.480+0000 7fc9b27fc640 1 -- 172.21.15.117:0/1335051543 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6061 (secure 0 0 0) 0x7fc9b417dc80 con 0x7fc9b40db580 2024-09-17T05:35:27.482 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:27.482 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":40,"btime":"2024-09-17T05:35:25:911092+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:25.911088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:27.482 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 40 2024-09-17T05:35:27.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.492+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9a0078b90 msgr2=0x7fc9a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.492+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9a0078b90 0x7fc9a007b0a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8000f60 tx=0x7fc9b8009040 comp rx=0 tx=0).stop 2024-09-17T05:35:27.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.492+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 msgr2=0x7fc9b40d3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.492+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 0x7fc9b40d3f30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc06bdf0 tx=0x7fc9bc06a0a0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 shutdown_connections 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9a0078b90 0x7fc9a007b0a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9b40db580 0x7fc9b40d3f30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9b40d5450 0x7fc9b40c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 --2- 172.21.15.117:0/1335051543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9b4033290 0x7fc9b4049c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 >> 172.21.15.117:0/1335051543 conn(0x7fc9b4050280 msgr2=0x7fc9b4035cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 shutdown_connections 2024-09-17T05:35:27.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.493+0000 7fc9c2a01640 1 -- 172.21.15.117:0/1335051543 wait complete. 2024-09-17T05:35:27.517 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-09-17T05:35:27.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-09-17T05:35:27.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:27.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.729+0000 7f2b01f8f640 1 -- 172.21.15.117:0/3802671380 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af40d2590 msgr2=0x7f2af4008990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.729+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/3802671380 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af40d2590 0x7f2af4008990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009830 tx=0x7f2af002a530 comp rx=0 tx=0).stop 2024-09-17T05:35:27.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.730+0000 7f2b01f8f640 1 -- 172.21.15.117:0/3802671380 shutdown_connections 2024-09-17T05:35:27.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.730+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/3802671380 >> v2:172.21.15.119:3300/0 conn(0x7f2af404d260 0x7f2af40def50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.730+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/3802671380 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af40d2590 0x7f2af4008990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.730+0000 7f2b01f8f640 1 -- 172.21.15.117:0/3802671380 >> 172.21.15.117:0/3802671380 conn(0x7f2af4050f50 msgr2=0x7f2af4051360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:27.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.734+0000 7f2b01f8f640 1 -- 172.21.15.117:0/3802671380 shutdown_connections 2024-09-17T05:35:27.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.737+0000 7f2b01f8f640 1 -- 172.21.15.117:0/3802671380 wait complete. 2024-09-17T05:35:27.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.740+0000 7f2b01f8f640 1 Processor -- start 2024-09-17T05:35:27.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.750+0000 7f2b01f8f640 1 -- start start 2024-09-17T05:35:27.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.752+0000 7f2b01f8f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af4008ed0 0x7f2af4049970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.752+0000 7f2b01f8f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2af404d260 0x7f2af4048950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b01f8f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 0x7f2af40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b00f8d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af4008ed0 0x7f2af4049970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b00f8d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af4008ed0 0x7f2af4049970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:36908/0 (socket says 172.21.15.117:36908) 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b0178e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 0x7f2af40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b0178e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 0x7f2af40cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:48184/0 (socket says 172.21.15.117:48184) 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b00f8d640 1 -- 172.21.15.117:0/2251596988 learned_addr learned my addr 172.21.15.117:0/2251596988 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2af40dfaf0 con 0x7f2af40d2590 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2af4008500 con 0x7f2af404d260 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2af40d5f20 con 0x7f2af4008ed0 2024-09-17T05:35:27.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.753+0000 7f2afbfff640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2af404d260 0x7f2af4048950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.754+0000 7f2b0178e640 1 -- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af4008ed0 msgr2=0x7f2af4049970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.754+0000 7f2b0178e640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af4008ed0 0x7f2af4049970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.754+0000 7f2b0178e640 1 -- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2af404d260 msgr2=0x7f2af4048950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:27.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.754+0000 7f2b0178e640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2af404d260 0x7f2af4048950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:27.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.754+0000 7f2b0178e640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af4049eb0 con 0x7f2af40d2590 2024-09-17T05:35:27.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.755+0000 7f2b0178e640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 0x7f2af40cc8a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0521d0 tx=0x7f2afc0519a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:27.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.756+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc097070 con 0x7f2af40d2590 2024-09-17T05:35:27.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.757+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2afc0753d0 con 0x7f2af40d2590 2024-09-17T05:35:27.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.757+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af404aef0 con 0x7f2af40d2590 2024-09-17T05:35:27.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.761+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af40d2b70 con 0x7f2af40d2590 2024-09-17T05:35:27.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.762+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0738f0 con 0x7f2af40d2590 2024-09-17T05:35:27.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.762+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2afc073b10 con 0x7f2af40d2590 2024-09-17T05:35:27.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.764+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af40d9280 con 0x7f2af40d2590 2024-09-17T05:35:27.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.777+0000 7f2af97fa640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2ae0078c90 0x7f2ae007b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:27.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.778+0000 7f2b00f8d640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2ae0078c90 0x7f2ae007b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:27.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.779+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f2afc11f060 con 0x7f2af40d2590 2024-09-17T05:35:27.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.780+0000 7f2b00f8d640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2ae0078c90 0x7f2ae007b1a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2af0005eb0 tx=0x7f2af0004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:27.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:27.781+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2afc11ab40 con 0x7f2af40d2590 2024-09-17T05:35:28.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.231+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2af4050d30 con 0x7f2af40d2590 2024-09-17T05:35:28.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.233+0000 7f2af97fa640 1 -- 172.21.15.117:0/2251596988 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6061 (secure 0 0 0) 0x7f2afc0e2650 con 0x7f2af40d2590 2024-09-17T05:35:28.237 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:28.237 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":40,"btime":"2024-09-17T05:35:25:911092+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:25.911088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:28.238 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 40 2024-09-17T05:35:28.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2ae0078c90 msgr2=0x7f2ae007b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:28.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2ae0078c90 0x7f2ae007b1a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2af0005eb0 tx=0x7f2af0004730 comp rx=0 tx=0).stop 2024-09-17T05:35:28.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 msgr2=0x7f2af40cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:28.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 0x7f2af40cc8a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0521d0 tx=0x7f2afc0519a0 comp rx=0 tx=0).stop 2024-09-17T05:35:28.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 shutdown_connections 2024-09-17T05:35:28.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2ae0078c90 0x7f2ae007b1a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:28.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2af40d2590 0x7f2af40cc8a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:28.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2af404d260 0x7f2af4048950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:28.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 --2- 172.21.15.117:0/2251596988 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2af4008ed0 0x7f2af4049970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:28.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.237+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 >> 172.21.15.117:0/2251596988 conn(0x7f2af4050f50 msgr2=0x7f2af40d0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:28.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.238+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 shutdown_connections 2024-09-17T05:35:28.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:28.238+0000 7f2b01f8f640 1 -- 172.21.15.117:0/2251596988 wait complete. 2024-09-17T05:35:28.266 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:35:25:911092+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5457": { "addr": "172.21.15.117:6837/1696536401", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 1696536401, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 1696536401, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5457, "incarnation": 40, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:35:25.911088+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5457 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6835/2865340338", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 2865340338, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 2865340338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 39, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5084, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:35:30.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:30.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.484+0000 7fb75affd640 1 -- 172.21.15.117:0/981568641 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74c0062b0 con 0x7fb744003a80 2024-09-17T05:35:30.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb7624a9640 1 -- 172.21.15.117:0/981568641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb744003a80 msgr2=0x7fb744003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:30.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb7624a9640 1 --2- 172.21.15.117:0/981568641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb744003a80 0x7fb744003ef0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0016c0 tx=0x7fb744005830 comp rx=0 tx=0).stop 2024-09-17T05:35:30.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb760a46640 1 -- 172.21.15.117:0/981568641 reap_dead start 2024-09-17T05:35:30.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb7624a9640 1 -- 172.21.15.117:0/981568641 shutdown_connections 2024-09-17T05:35:30.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb7624a9640 1 -- 172.21.15.117:0/981568641 >> 172.21.15.117:0/981568641 conn(0x7fb75c06d190 msgr2=0x7fb75c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:30.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb7624a9640 1 -- 172.21.15.117:0/981568641 shutdown_connections 2024-09-17T05:35:30.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.485+0000 7fb7624a9640 1 -- 172.21.15.117:0/981568641 wait complete. 2024-09-17T05:35:30.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.488+0000 7fb7624a9640 1 Processor -- start 2024-09-17T05:35:30.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.488+0000 7fb7624a9640 1 -- start start 2024-09-17T05:35:30.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.490+0000 7fb7624a9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb75c14df10 0x7fb75c146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:30.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb7624a9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 0x7fb75c1f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:30.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb7624a9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb75c1f5f20 0x7fb75c156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:30.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb7624a9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c0737d0 con 0x7fb75c0c30b0 2024-09-17T05:35:30.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb7624a9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c074dc0 con 0x7fb75c14df10 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb7624a9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb75c073950 con 0x7fb75c1f5f20 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb75b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 0x7fb75c1f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb75b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 0x7fb75c1f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:48214/0 (socket says 172.21.15.117:48214) 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.491+0000 7fb75b7fe640 1 -- 172.21.15.117:0/1467359866 learned_addr learned my addr 172.21.15.117:0/1467359866 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.492+0000 7fb75b7fe640 1 -- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb75c1f5f20 msgr2=0x7fb75c156fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.492+0000 7fb75b7fe640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb75c1f5f20 0x7fb75c156fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.492+0000 7fb75b7fe640 1 -- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb75c14df10 msgr2=0x7fb75c146fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.492+0000 7fb75b7fe640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb75c14df10 0x7fb75c146fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:30.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.492+0000 7fb75b7fe640 1 -- 172.21.15.117:0/1467359866 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75c1e1f80 con 0x7fb75c0c30b0 2024-09-17T05:35:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.492+0000 7fb75b7fe640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 0x7fb75c1f59e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb75000e8b0 tx=0x7fb75000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.493+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb750015070 con 0x7fb75c0c30b0 2024-09-17T05:35:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.493+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb75c1e0b90 con 0x7fb75c0c30b0 2024-09-17T05:35:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.493+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb75c157b30 con 0x7fb75c0c30b0 2024-09-17T05:35:30.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.495+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb75c148f10 con 0x7fb75c0c30b0 2024-09-17T05:35:30.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.504+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb75000cad0 con 0x7fb75c0c30b0 2024-09-17T05:35:30.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.504+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb750011460 con 0x7fb75c0c30b0 2024-09-17T05:35:30.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.507+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb750018070 con 0x7fb75c0c30b0 2024-09-17T05:35:30.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.508+0000 7fb758ff9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb748078c00 0x7fb74807b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:30.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.508+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fb75009bb40 con 0x7fb75c0c30b0 2024-09-17T05:35:30.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.509+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb75c148f10 con 0x7fb75c0c30b0 2024-09-17T05:35:30.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.509+0000 7fb75bfff640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb748078c00 0x7fb74807b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:30.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.516+0000 7fb75bfff640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb748078c00 0x7fb74807b110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb74c032000 tx=0x7fb74c001190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:30.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.966+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb75c0c9af0 con 0x7fb75c0c30b0 2024-09-17T05:35:30.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.968+0000 7fb758ff9640 1 -- 172.21.15.117:0/1467359866 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6061 (secure 0 0 0) 0x7fb750063a90 con 0x7fb75c0c30b0 2024-09-17T05:35:30.972 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:30.973 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":40,"btime":"2024-09-17T05:35:25:911092+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:25.911088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:30.973 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 40 2024-09-17T05:35:30.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.973+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb748078c00 msgr2=0x7fb74807b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:30.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.973+0000 7fb7624a9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb748078c00 0x7fb74807b110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb74c032000 tx=0x7fb74c001190 comp rx=0 tx=0).stop 2024-09-17T05:35:30.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.973+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 msgr2=0x7fb75c1f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:30.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.973+0000 7fb7624a9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 0x7fb75c1f59e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb75000e8b0 tx=0x7fb75000ed70 comp rx=0 tx=0).stop 2024-09-17T05:35:30.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.979+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 shutdown_connections 2024-09-17T05:35:30.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.979+0000 7fb7624a9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb748078c00 0x7fb74807b110 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:30.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.979+0000 7fb7624a9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb75c1f5f20 0x7fb75c156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:30.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.979+0000 7fb7624a9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c0c30b0 0x7fb75c1f59e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:30.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.979+0000 7fb7624a9640 1 --2- 172.21.15.117:0/1467359866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb75c14df10 0x7fb75c146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:30.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.979+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 >> 172.21.15.117:0/1467359866 conn(0x7fb75c06d190 msgr2=0x7fb75c0cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:30.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.980+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 shutdown_connections 2024-09-17T05:35:30.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:30.980+0000 7fb7624a9640 1 -- 172.21.15.117:0/1467359866 wait complete. 2024-09-17T05:35:31.002 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:33.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 -- 172.21.15.117:0/4008919421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c52d0 msgr2=0x7f563c0c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 --2- 172.21.15.117:0/4008919421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c52d0 0x7f563c0c7780 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f563c0c7ed0 tx=0x7f563002a530 comp rx=0 tx=0).stop 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 -- 172.21.15.117:0/4008919421 shutdown_connections 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 --2- 172.21.15.117:0/4008919421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c52d0 0x7f563c0c7780 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 --2- 172.21.15.117:0/4008919421 >> v2:172.21.15.119:3300/0 conn(0x7f563c0c4c40 0x7f563c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 -- 172.21.15.117:0/4008919421 >> 172.21.15.117:0/4008919421 conn(0x7f563c06d190 msgr2=0x7f563c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.183+0000 7f5642762640 1 -- 172.21.15.117:0/4008919421 shutdown_connections 2024-09-17T05:35:33.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.184+0000 7f5642762640 1 -- 172.21.15.117:0/4008919421 wait complete. 2024-09-17T05:35:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.187+0000 7f5642762640 1 Processor -- start 2024-09-17T05:35:33.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.188+0000 7f5642762640 1 -- start start 2024-09-17T05:35:33.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.191+0000 7f5642762640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f563c0c3820 0x7f563c0c1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:33.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.191+0000 7f5642762640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 0x7f563c0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:33.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.191+0000 7f5642762640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c0c52d0 0x7f563c0cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:33.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.191+0000 7f5642762640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f563c0c6f80 con 0x7f563c0c4c40 2024-09-17T05:35:33.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.191+0000 7f5642762640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f563c0c6ca0 con 0x7f563c0c52d0 2024-09-17T05:35:33.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.191+0000 7f5642762640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f563c0c9360 con 0x7f563c0c3820 2024-09-17T05:35:33.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.192+0000 7f563b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 0x7f563c0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:33.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.193+0000 7f563bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f563c0c3820 0x7f563c0c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.194+0000 7f563b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 0x7f563c0caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:48224/0 (socket says 172.21.15.117:48224) 2024-09-17T05:35:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.194+0000 7f563b7fe640 1 -- 172.21.15.117:0/3155499165 learned_addr learned my addr 172.21.15.117:0/3155499165 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:33.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.195+0000 7f5640cff640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c0c52d0 0x7f563c0cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.195+0000 7f563b7fe640 1 -- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f563c0c3820 msgr2=0x7f563c0c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.195+0000 7f563b7fe640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f563c0c3820 0x7f563c0c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.195+0000 7f563b7fe640 1 -- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c0c52d0 msgr2=0x7f563c0cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.195+0000 7f563b7fe640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c0c52d0 0x7f563c0cb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f563b7fe640 1 -- 172.21.15.117:0/3155499165 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c0cba40 con 0x7f563c0c4c40 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f563bfff640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f563c0c3820 0x7f563c0c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f563b7fe640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 0x7f563c0caeb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f56300098f0 tx=0x7f5630033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630005c90 con 0x7f563c0c4c40 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5630031b60 con 0x7f563c0c4c40 2024-09-17T05:35:33.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630005020 con 0x7f563c0c4c40 2024-09-17T05:35:33.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.196+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c0ccc60 con 0x7f563c0c4c40 2024-09-17T05:35:33.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.197+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c068f90 con 0x7f563c0c4c40 2024-09-17T05:35:33.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.199+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563c0c9800 con 0x7f563c0c4c40 2024-09-17T05:35:33.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.199+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f56300051c0 con 0x7f563c0c4c40 2024-09-17T05:35:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.200+0000 7f5638ff9640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5628078be0 0x7f562807b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.200+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f56300bfca0 con 0x7f563c0c4c40 2024-09-17T05:35:33.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.205+0000 7f563bfff640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5628078be0 0x7f562807b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.208+0000 7f563bfff640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5628078be0 0x7f562807b0f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f562c009ab0 tx=0x7f562c005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:33.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.209+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f56300875a0 con 0x7f563c0c4c40 2024-09-17T05:35:33.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.587+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f563c14ec50 con 0x7f563c0c4c40 2024-09-17T05:35:33.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.588+0000 7f5638ff9640 1 -- 172.21.15.117:0/3155499165 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6061 (secure 0 0 0) 0x7f563008bfd0 con 0x7f563c0c4c40 2024-09-17T05:35:33.590 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:33.590 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":40,"btime":"2024-09-17T05:35:25:911092+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:25.911088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:replay","state_seq":1,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:33.590 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 40 2024-09-17T05:35:33.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.591+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5628078be0 msgr2=0x7f562807b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.591+0000 7f5642762640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5628078be0 0x7f562807b0f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f562c009ab0 tx=0x7f562c005f90 comp rx=0 tx=0).stop 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.591+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 msgr2=0x7f563c0caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.591+0000 7f5642762640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 0x7f563c0caeb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f56300098f0 tx=0x7f5630033c20 comp rx=0 tx=0).stop 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 shutdown_connections 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5628078be0 0x7f562807b0f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c0c52d0 0x7f563c0cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f563c0c4c40 0x7f563c0caeb0 secure :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f56300098f0 tx=0x7f5630033c20 comp rx=0 tx=0).stop 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 --2- 172.21.15.117:0/3155499165 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f563c0c3820 0x7f563c0c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:33.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 >> 172.21.15.117:0/3155499165 conn(0x7f563c06d190 msgr2=0x7f563c074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:33.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.592+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 shutdown_connections 2024-09-17T05:35:33.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:33.593+0000 7f5642762640 1 -- 172.21.15.117:0/3155499165 wait complete. 2024-09-17T05:35:33.613 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:35.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:35.842 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.840+0000 7fec4dd2b640 1 -- 172.21.15.117:0/3922857445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 msgr2=0x7fec48074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.840+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/3922857445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 0x7fec48074b60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fec480724d0 tx=0x7fec4002b820 comp rx=0 tx=0).stop 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.840+0000 7fec4dd2b640 1 -- 172.21.15.117:0/3922857445 shutdown_connections 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.840+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/3922857445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 0x7fec48074b60 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.840+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/3922857445 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fec480c92c0 0x7fec480c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.841+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/3922857445 >> v2:172.21.15.119:3300/0 conn(0x7fec4808c4c0 0x7fec480c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.841+0000 7fec4dd2b640 1 -- 172.21.15.117:0/3922857445 >> 172.21.15.117:0/3922857445 conn(0x7fec4806d190 msgr2=0x7fec4806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.841+0000 7fec4dd2b640 1 -- 172.21.15.117:0/3922857445 shutdown_connections 2024-09-17T05:35:35.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.841+0000 7fec4dd2b640 1 -- 172.21.15.117:0/3922857445 wait complete. 2024-09-17T05:35:35.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.844+0000 7fec4dd2b640 1 Processor -- start 2024-09-17T05:35:35.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.845+0000 7fec4dd2b640 1 -- start start 2024-09-17T05:35:35.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.847+0000 7fec4dd2b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 0x7fec480cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:35.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.847+0000 7fec4dd2b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 0x7fec4815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:35.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.847+0000 7fec4dd2b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fec480c92c0 0x7fec48155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:35.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.847+0000 7fec4dd2b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec480c8b80 con 0x7fec48074750 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.847+0000 7fec4dd2b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fec4816b5d0 con 0x7fec4808c4c0 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.847+0000 7fec4dd2b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fec480c8d00 con 0x7fec480c92c0 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 0x7fec4815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 0x7fec4815b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:33248/0 (socket says 172.21.15.117:33248) 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 -- 172.21.15.117:0/784202150 learned_addr learned my addr 172.21.15.117:0/784202150 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 -- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fec480c92c0 msgr2=0x7fec48155e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fec480c92c0 0x7fec48155e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 -- 172.21.15.117:0/784202150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 msgr2=0x7fec480cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 0x7fec480cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.848+0000 7fec46ffd640 1 -- 172.21.15.117:0/784202150 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec48158b70 con 0x7fec4808c4c0 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.849+0000 7fec46ffd640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 0x7fec4815b410 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00d740 tx=0x7fec3c00dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:35.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.849+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c017070 con 0x7fec4808c4c0 2024-09-17T05:35:35.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.850+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4815be60 con 0x7fec4808c4c0 2024-09-17T05:35:35.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.850+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec480cbad0 con 0x7fec4808c4c0 2024-09-17T05:35:35.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.850+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec3c009d20 con 0x7fec4808c4c0 2024-09-17T05:35:35.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.850+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c0106c0 con 0x7fec4808c4c0 2024-09-17T05:35:35.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.853+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fec3c01a070 con 0x7fec4808c4c0 2024-09-17T05:35:35.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.854+0000 7fec2ffff640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fec34078c90 0x7fec3407b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:35.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.854+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fec3c09b0b0 con 0x7fec4808c4c0 2024-09-17T05:35:35.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.853+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec28005290 con 0x7fec4808c4c0 2024-09-17T05:35:35.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.861+0000 7fec477fe640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fec34078c90 0x7fec3407b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:35.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.865+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec3c063120 con 0x7fec4808c4c0 2024-09-17T05:35:35.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:35.869+0000 7fec477fe640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fec34078c90 0x7fec3407b1a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fec38005fd0 tx=0x7fec380094c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:36.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.341+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec28005580 con 0x7fec4808c4c0 2024-09-17T05:35:36.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.342+0000 7fec2ffff640 1 -- 172.21.15.117:0/784202150 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6065 (secure 0 0 0) 0x7fec3c062a20 con 0x7fec4808c4c0 2024-09-17T05:35:36.344 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:36.344 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":41,"btime":"2024-09-17T05:35:35:090150+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:34.589631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:36.344 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 41 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.346+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fec34078c90 msgr2=0x7fec3407b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.346+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fec34078c90 0x7fec3407b1a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fec38005fd0 tx=0x7fec380094c0 comp rx=0 tx=0).stop 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.346+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 msgr2=0x7fec4815b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.346+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 0x7fec4815b410 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00d740 tx=0x7fec3c00dc00 comp rx=0 tx=0).stop 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.347+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 shutdown_connections 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.347+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fec34078c90 0x7fec3407b1a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:36.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.347+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fec480c92c0 0x7fec48155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:36.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.347+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4808c4c0 0x7fec4815b410 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:36.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.347+0000 7fec4dd2b640 1 --2- 172.21.15.117:0/784202150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec48074750 0x7fec480cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:36.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.347+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 >> 172.21.15.117:0/784202150 conn(0x7fec4806d190 msgr2=0x7fec48068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:36.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.351+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 shutdown_connections 2024-09-17T05:35:36.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:36.351+0000 7fec4dd2b640 1 -- 172.21.15.117:0/784202150 wait complete. 2024-09-17T05:35:36.374 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:38.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:38.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.567+0000 7f2fc4bca640 1 -- 172.21.15.117:0/1115430665 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8031810 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.568+0000 7f2fc7e30640 1 -- 172.21.15.117:0/1115430665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc00c6ca0 msgr2=0x7f2fc015ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:38.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.568+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/1115430665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc015ae60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009b60 tx=0x7f2fb802a8b0 comp rx=0 tx=0).stop 2024-09-17T05:35:38.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.568+0000 7f2fc7e30640 1 -- 172.21.15.117:0/1115430665 shutdown_connections 2024-09-17T05:35:38.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.568+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/1115430665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc015ae60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:38.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.568+0000 7f2fc7e30640 1 -- 172.21.15.117:0/1115430665 >> 172.21.15.117:0/1115430665 conn(0x7f2fc006d190 msgr2=0x7f2fc006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:38.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.571+0000 7f2fc7e30640 1 -- 172.21.15.117:0/1115430665 shutdown_connections 2024-09-17T05:35:38.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.575+0000 7f2fc7e30640 1 -- 172.21.15.117:0/1115430665 wait complete. 2024-09-17T05:35:38.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.578+0000 7f2fc7e30640 1 Processor -- start 2024-09-17T05:35:38.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.578+0000 7f2fc7e30640 1 -- start start 2024-09-17T05:35:38.581 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.580+0000 7f2fc7e30640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc0072e10 0x7f2fc01f91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:38.581 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.580+0000 7f2fc7e30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc01f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:38.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc53cb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc01f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:38.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc53cb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc01f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41204/0 (socket says 172.21.15.117:41204) 2024-09-17T05:35:38.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc53cb640 1 -- 172.21.15.117:0/457622918 learned_addr learned my addr 172.21.15.117:0/457622918 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:38.586 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fc0157ea0 0x7f2fc0150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc5bcc640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc0072e10 0x7f2fc01f91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc00c67f0 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc00c3820 con 0x7f2fc0072e10 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc00c9b20 con 0x7f2fc0157ea0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.581+0000 7f2fc63cd640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fc0157ea0 0x7f2fc0150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.582+0000 7f2fc53cb640 1 -- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fc0157ea0 msgr2=0x7f2fc0150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.582+0000 7f2fc53cb640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fc0157ea0 0x7f2fc0150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.582+0000 7f2fc53cb640 1 -- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc0072e10 msgr2=0x7f2fc01f91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.582+0000 7f2fc53cb640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc0072e10 0x7f2fc01f91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.582+0000 7f2fc53cb640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc01f6bc0 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.583+0000 7f2fc53cb640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc01f6680 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000c760 tx=0x7f2fb000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.584+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb0005270 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.584+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fb0005410 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.587+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc01f9920 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.589+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc015edc0 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.590+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb000a8a0 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.592+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2fb000aac0 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.594+0000 7f2fb67fc640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fa4078be0 0x7f2fa407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:38.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.594+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa8005290 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.595+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f2fb009a6b0 con 0x7f2fc00c6ca0 2024-09-17T05:35:38.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.596+0000 7f2fc5bcc640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fa4078be0 0x7f2fa407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:38.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.598+0000 7f2fc5bcc640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fa4078be0 0x7f2fa407b0f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:38.604+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2fb0062720 con 0x7f2fc00c6ca0 2024-09-17T05:35:39.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.041+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fa8005850 con 0x7f2fc00c6ca0 2024-09-17T05:35:39.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.043+0000 7f2fb67fc640 1 -- 172.21.15.117:0/457622918 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6065 (secure 0 0 0) 0x7f2fb0062020 con 0x7f2fc00c6ca0 2024-09-17T05:35:39.045 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:39.045 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":41,"btime":"2024-09-17T05:35:35:090150+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:34.589631+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:39.045 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 41 2024-09-17T05:35:39.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.047+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fa4078be0 msgr2=0x7f2fa407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:39.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.047+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fa4078be0 0x7f2fa407b0f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc009040 comp rx=0 tx=0).stop 2024-09-17T05:35:39.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.047+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 msgr2=0x7f2fc01f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.047+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc01f6680 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fb000c760 tx=0x7f2fb000cc20 comp rx=0 tx=0).stop 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.048+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 shutdown_connections 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.048+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2fa4078be0 0x7f2fa407b0f0 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc009040 comp rx=0 tx=0).stop 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.048+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2fc0157ea0 0x7f2fc0150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.048+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2fc00c6ca0 0x7f2fc01f6680 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.048+0000 7f2fc7e30640 1 --2- 172.21.15.117:0/457622918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fc0072e10 0x7f2fc01f91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.048+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 >> 172.21.15.117:0/457622918 conn(0x7f2fc006d190 msgr2=0x7f2fc014dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:39.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.049+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 shutdown_connections 2024-09-17T05:35:39.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:39.049+0000 7f2fc7e30640 1 -- 172.21.15.117:0/457622918 wait complete. 2024-09-17T05:35:39.069 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:41.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:41.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.295+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3970645539 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe784004800 msgr2=0x7fe784004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:41.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.295+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3970645539 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe784004800 0x7fe784004c10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe79c073500 tx=0x7fe784005830 comp rx=0 tx=0).stop 2024-09-17T05:35:41.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.295+0000 7fe79b7fe640 1 -- 172.21.15.117:0/3970645539 reap_dead start 2024-09-17T05:35:41.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.296+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3970645539 shutdown_connections 2024-09-17T05:35:41.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.296+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3970645539 >> 172.21.15.117:0/3970645539 conn(0x7fe79c0c9a20 msgr2=0x7fe79c0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:41.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.300+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3970645539 shutdown_connections 2024-09-17T05:35:41.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.300+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3970645539 wait complete. 2024-09-17T05:35:41.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.303+0000 7fe7a19f1640 1 Processor -- start 2024-09-17T05:35:41.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.304+0000 7fe7a19f1640 1 -- start start 2024-09-17T05:35:41.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.306+0000 7fe7a19f1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe79c151330 0x7fe79c0aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:41.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.306+0000 7fe7a19f1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 0x7fe79c15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:41.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.306+0000 7fe7a19f1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe79c0b1fb0 0x7fe79c15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:41.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.306+0000 7fe79a7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 0x7fe79c15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:41.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.306+0000 7fe79a7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 0x7fe79c15a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41212/0 (socket says 172.21.15.117:41212) 2024-09-17T05:35:41.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.311+0000 7fe79a7fc640 1 -- 172.21.15.117:0/3760463997 learned_addr learned my addr 172.21.15.117:0/3760463997 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.311+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c157af0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.311+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe79c074370 con 0x7fe79c0b1fb0 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.311+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe79c157dd0 con 0x7fe79c151330 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.312+0000 7fe79a7fc640 1 -- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe79c151330 msgr2=0x7fe79c0aaa90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.312+0000 7fe79a7fc640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe79c151330 0x7fe79c0aaa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.312+0000 7fe79a7fc640 1 -- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe79c0b1fb0 msgr2=0x7fe79c15aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.312+0000 7fe79a7fc640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe79c0b1fb0 0x7fe79c15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.312+0000 7fe79a7fc640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79c1edb70 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.312+0000 7fe79a7fc640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 0x7fe79c15a550 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe79400ace0 tx=0x7fe794032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:41.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.314+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794035070 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.314+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe79403b670 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.314+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79400a840 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.316+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79c0b2500 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.316+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79c151db0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.318+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe79c0c94d0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.320+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe79403bac0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.327+0000 7fe783fff640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe788078c90 0x7fe78807b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:41.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.327+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fe7940bc770 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.332+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe7940847e0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.335+0000 7fe79affd640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe788078c90 0x7fe78807b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:41.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.338+0000 7fe79affd640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe788078c90 0x7fe78807b1a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0095f0 tx=0x7fe78c006bd0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:41.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.787+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe79c0aafd0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.788+0000 7fe783fff640 1 -- 172.21.15.117:0/3760463997 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6062 (secure 0 0 0) 0x7fe7940840e0 con 0x7fe79c0b1ba0 2024-09-17T05:35:41.791 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:41.791 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":42,"btime":"2024-09-17T05:35:40:125979+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:40.102576+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:41.791 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 42 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe788078c90 msgr2=0x7fe78807b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe788078c90 0x7fe78807b1a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0095f0 tx=0x7fe78c006bd0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 msgr2=0x7fe79c15a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 0x7fe79c15a550 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe79400ace0 tx=0x7fe794032e70 comp rx=0 tx=0).stop 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 shutdown_connections 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fe788078c90 0x7fe78807b1a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe79c0b1fb0 0x7fe79c15aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe79c0b1ba0 0x7fe79c15a550 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 --2- 172.21.15.117:0/3760463997 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe79c151330 0x7fe79c0aaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:41.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.794+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 >> 172.21.15.117:0/3760463997 conn(0x7fe79c0c9a20 msgr2=0x7fe79c0c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:41.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.795+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 shutdown_connections 2024-09-17T05:35:41.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:41.795+0000 7fe7a19f1640 1 -- 172.21.15.117:0/3760463997 wait complete. 2024-09-17T05:35:41.824 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "btime": "2024-09-17T05:35:40:125979+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T05:27:26.003433+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5457": { "addr": "172.21.15.117:6837/1696536401", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6836", "nonce": 1696536401, "type": "v2" }, { "addr": "172.21.15.117:6837", "nonce": 1696536401, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5457, "incarnation": 40, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:35:40.102576+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5457 } } } ], "standbys": [ { "addr": "172.21.15.119:6837/3540182507", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6836", "nonce": 3540182507, "type": "v2" }, { "addr": "172.21.15.119:6837", "nonce": 3540182507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4337, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6833/2497891489", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6832", "nonce": 2497891489, "type": "v2" }, { "addr": "172.21.15.117:6833", "nonce": 2497891489, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4416, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.117:6835/2865340338", "addrs": { "addrvec": [ { "addr": "172.21.15.117:6834", "nonce": 2865340338, "type": "v2" }, { "addr": "172.21.15.117:6835", "nonce": 2865340338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 39, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5084, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.119:6835/2852680530", "addrs": { "addrvec": [ { "addr": "172.21.15.119:6834", "nonce": 2852680530, "type": "v2" }, { "addr": "172.21.15.119:6835", "nonce": 2852680530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 32, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5263, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:35:43.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:44.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.039+0000 7f5870e98640 1 -- 172.21.15.117:0/3544420757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c1516d0 msgr2=0x7f586c157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:44.040 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.039+0000 7f5870e98640 1 --2- 172.21.15.117:0/3544420757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c1516d0 0x7f586c157df0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5864005fd0 tx=0x7f5864032d20 comp rx=0 tx=0).stop 2024-09-17T05:35:44.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.039+0000 7f5870e98640 1 -- 172.21.15.117:0/3544420757 shutdown_connections 2024-09-17T05:35:44.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.039+0000 7f5870e98640 1 --2- 172.21.15.117:0/3544420757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c1516d0 0x7f586c157df0 secure :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5864005fd0 tx=0x7f5864032d20 comp rx=0 tx=0).stop 2024-09-17T05:35:44.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.039+0000 7f5870e98640 1 -- 172.21.15.117:0/3544420757 >> 172.21.15.117:0/3544420757 conn(0x7f586c06cda0 msgr2=0x7f586c06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:44.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.040+0000 7f5870e98640 1 -- 172.21.15.117:0/3544420757 shutdown_connections 2024-09-17T05:35:44.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.041+0000 7f5870e98640 1 -- 172.21.15.117:0/3544420757 wait complete. 2024-09-17T05:35:44.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.044+0000 7f5870e98640 1 Processor -- start 2024-09-17T05:35:44.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.044+0000 7f5870e98640 1 -- start start 2024-09-17T05:35:44.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f5870e98640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c0c7890 0x7f586c1fe570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:44.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f5870e98640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f586c0cc910 0x7f586c148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:44.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f5870e98640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 0x7f586c14f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:44.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f5870e98640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1575f0 con 0x7f586c0c7890 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f5870e98640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f586c157960 con 0x7f586c14eef0 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f5870e98640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f586c158330 con 0x7f586c0cc910 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f586affd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f586c0cc910 0x7f586c148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f586bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 0x7f586c14f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f586bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 0x7f586c14f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34542/0 (socket says 172.21.15.117:34542) 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f586bfff640 1 -- 172.21.15.117:0/1762939362 learned_addr learned my addr 172.21.15.117:0/1762939362 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.047+0000 7f586b7fe640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c0c7890 0x7f586c1fe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.048+0000 7f586bfff640 1 -- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f586c0cc910 msgr2=0x7f586c148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.048+0000 7f586bfff640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f586c0cc910 0x7f586c148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.048+0000 7f586bfff640 1 -- 172.21.15.117:0/1762939362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c0c7890 msgr2=0x7f586c1fe570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.048+0000 7f586bfff640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c0c7890 0x7f586c1fe570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.048+0000 7f586bfff640 1 -- 172.21.15.117:0/1762939362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c14e030 con 0x7f586c14eef0 2024-09-17T05:35:44.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.048+0000 7f586bfff640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 0x7f586c14f300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f586000b440 tx=0x7f586000b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:44.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.049+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5860015020 con 0x7f586c14eef0 2024-09-17T05:35:44.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.049+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5860011940 con 0x7f586c14eef0 2024-09-17T05:35:44.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.049+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5860011d90 con 0x7f586c14eef0 2024-09-17T05:35:44.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.049+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586c154be0 con 0x7f586c14eef0 2024-09-17T05:35:44.051 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.049+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586c14dd50 con 0x7f586c14eef0 2024-09-17T05:35:44.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.051+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586c0c8400 con 0x7f586c14eef0 2024-09-17T05:35:44.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.053+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5860018020 con 0x7f586c14eef0 2024-09-17T05:35:44.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.054+0000 7f5853fff640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5858078c90 0x7f585807b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:44.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.054+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f586009aed0 con 0x7f586c14eef0 2024-09-17T05:35:44.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.055+0000 7f586b7fe640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5858078c90 0x7f585807b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:44.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.062+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f586001f020 con 0x7f586c14eef0 2024-09-17T05:35:44.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.063+0000 7f586b7fe640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5858078c90 0x7f585807b1a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f586403c000 tx=0x7f5864004ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:44.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.508+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f586c0b29e0 con 0x7f586c14eef0 2024-09-17T05:35:44.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.509+0000 7f5853fff640 1 -- 172.21.15.117:0/1762939362 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6062 (secure 0 0 0) 0x7f58600628f0 con 0x7f586c14eef0 2024-09-17T05:35:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:44.511 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":42,"btime":"2024-09-17T05:35:40:125979+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:40.102576+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-09-17T05:35:44.511 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 42 2024-09-17T05:35:44.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.513+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5858078c90 msgr2=0x7f585807b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:44.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.513+0000 7f5870e98640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5858078c90 0x7f585807b1a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f586403c000 tx=0x7f5864004ab0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.513+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 msgr2=0x7f586c14f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:44.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.513+0000 7f5870e98640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 0x7f586c14f300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f586000b440 tx=0x7f586000b900 comp rx=0 tx=0).stop 2024-09-17T05:35:44.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.515+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 shutdown_connections 2024-09-17T05:35:44.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.515+0000 7f5870e98640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5858078c90 0x7f585807b1a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.515+0000 7f5870e98640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f586c14eef0 0x7f586c14f300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.515+0000 7f5870e98640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f586c0cc910 0x7f586c148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.515+0000 7f5870e98640 1 --2- 172.21.15.117:0/1762939362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f586c0c7890 0x7f586c1fe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:44.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.515+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 >> 172.21.15.117:0/1762939362 conn(0x7f586c06cda0 msgr2=0x7f586c156b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:44.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.517+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 shutdown_connections 2024-09-17T05:35:44.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:44.517+0000 7f5870e98640 1 -- 172.21.15.117:0/1762939362 wait complete. 2024-09-17T05:35:44.537 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-09-17T05:35:46.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:46.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.759+0000 7f1935f4b640 1 -- 172.21.15.117:0/3163611932 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f192804bbd0 msgr2=0x7f19280d5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:46.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.759+0000 7f1935f4b640 1 --2- 172.21.15.117:0/3163611932 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f192804bbd0 0x7f19280d5330 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f19300524b0 tx=0x7f19300a7390 comp rx=0 tx=0).stop 2024-09-17T05:35:46.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.759+0000 7f1935f4b640 1 -- 172.21.15.117:0/3163611932 shutdown_connections 2024-09-17T05:35:46.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.759+0000 7f1935f4b640 1 --2- 172.21.15.117:0/3163611932 >> v2:172.21.15.117:3300/0 conn(0x7f19280063a0 0x7f19280de5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:46.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.759+0000 7f1935f4b640 1 --2- 172.21.15.117:0/3163611932 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f192804bbd0 0x7f19280d5330 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:46.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.759+0000 7f1935f4b640 1 -- 172.21.15.117:0/3163611932 >> 172.21.15.117:0/3163611932 conn(0x7f19280506d0 msgr2=0x7f1928050ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:46.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.761+0000 7f1935f4b640 1 -- 172.21.15.117:0/3163611932 shutdown_connections 2024-09-17T05:35:46.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.761+0000 7f1935f4b640 1 -- 172.21.15.117:0/3163611932 wait complete. 2024-09-17T05:35:46.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.764+0000 7f1935f4b640 1 Processor -- start 2024-09-17T05:35:46.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.764+0000 7f1935f4b640 1 -- start start 2024-09-17T05:35:46.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.766+0000 7f1935f4b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1928005f90 0x7f1928182c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:46.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.766+0000 7f1935f4b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 0x7f19280467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:46.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.766+0000 7f1935f4b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f192804bbd0 0x7f19280ddca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:46.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.766+0000 7f1935f4b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1928007a40 con 0x7f19280063a0 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.766+0000 7f1935f4b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1928007760 con 0x7f192804bbd0 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.766+0000 7f1935f4b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f192804c9a0 con 0x7f1928005f90 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f1934f49640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1928005f90 0x7f1928182c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 0x7f19280467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 0x7f19280467a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44884/0 (socket says 172.21.15.117:44884) 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 -- 172.21.15.117:0/3961745944 learned_addr learned my addr 172.21.15.117:0/3961745944 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f193574a640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f192804bbd0 0x7f19280ddca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 -- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1928005f90 msgr2=0x7f1928182c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1928005f90 0x7f1928182c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 -- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f192804bbd0 msgr2=0x7f19280ddca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f192804bbd0 0x7f19280ddca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.767+0000 7f192ffff640 1 -- 172.21.15.117:0/3961745944 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19280c4160 con 0x7f19280063a0 2024-09-17T05:35:46.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.768+0000 7f192ffff640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 0x7f19280467a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c760 tx=0x7f191c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:46.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.768+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c006a90 con 0x7f19280063a0 2024-09-17T05:35:46.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.768+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f191c017070 con 0x7f19280063a0 2024-09-17T05:35:46.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.768+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c007030 con 0x7f19280063a0 2024-09-17T05:35:46.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.769+0000 7f1935f4b640 1 -- 172.21.15.117:0/3961745944 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1928183150 con 0x7f19280063a0 2024-09-17T05:35:46.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.769+0000 7f1935f4b640 1 -- 172.21.15.117:0/3961745944 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19280c2e30 con 0x7f19280063a0 2024-09-17T05:35:46.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.777+0000 7f1935f4b640 1 -- 172.21.15.117:0/3961745944 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1910005290 con 0x7f19280063a0 2024-09-17T05:35:46.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.777+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f191c0071d0 con 0x7f19280063a0 2024-09-17T05:35:46.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.778+0000 7f192d7fa640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1914078c60 0x7f191407b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:46.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.784+0000 7f1934f49640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1914078c60 0x7f191407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:46.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.791+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f191c09adb0 con 0x7f19280063a0 2024-09-17T05:35:46.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.796+0000 7f1934f49640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1914078c60 0x7f191407b170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f19280d58c0 tx=0x7f1930073040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:46.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:46.804+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f19280c4160 con 0x7f19280063a0 2024-09-17T05:35:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.260+0000 7f1935f4b640 1 -- 172.21.15.117:0/3961745944 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1910005580 con 0x7f19280063a0 2024-09-17T05:35:47.268 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:47.268 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":43,"btime":"2024-09-17T05:35:45:569662+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:45.569656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:active","state_seq":20,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5457,"qdb_cluster":[5457]},"id":1}]} 2024-09-17T05:35:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.264+0000 7f192d7fa640 1 -- 172.21.15.117:0/3961745944 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6069 (secure 0 0 0) 0x7f1928007200 con 0x7f19280063a0 2024-09-17T05:35:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 43 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 -- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1914078c60 msgr2=0x7f191407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1914078c60 0x7f191407b170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f19280d58c0 tx=0x7f1930073040 comp rx=0 tx=0).stop 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 -- 172.21.15.117:0/3961745944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 msgr2=0x7f19280467a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 0x7f19280467a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c760 tx=0x7f191c00cc20 comp rx=0 tx=0).stop 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 -- 172.21.15.117:0/3961745944 shutdown_connections 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1914078c60 0x7f191407b170 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f192804bbd0 0x7f19280ddca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19280063a0 0x7f19280467a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 --2- 172.21.15.117:0/3961745944 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1928005f90 0x7f1928182c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:47.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 -- 172.21.15.117:0/3961745944 >> 172.21.15.117:0/3961745944 conn(0x7f19280506d0 msgr2=0x7f19280d2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:47.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.268+0000 7f191affd640 1 -- 172.21.15.117:0/3961745944 shutdown_connections 2024-09-17T05:35:47.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:47.269+0000 7f191affd640 1 -- 172.21.15.117:0/3961745944 wait complete. 2024-09-17T05:35:47.289 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=4440 2024-09-17T05:35:47.290 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.f 2024-09-17T05:35:53.217 INFO:tasks.ceph.mds.e.smithi117.stderr:2024-09-17T05:35:53.216+0000 7fcfecb42640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:35:55.291 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-09-17T05:35:55.291 INFO:tasks.ceph.mds.f:Restarting daemon 2024-09-17T05:35:55.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-09-17T05:35:55.297 INFO:tasks.ceph.mds.f:Started 2024-09-17T05:35:55.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.478+0000 7f167ad6a640 1 -- 172.21.15.117:0/4199998724 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c0df7c0 msgr2=0x7f166c007770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.478+0000 7f167ad6a640 1 --2- 172.21.15.117:0/4199998724 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c0df7c0 0x7f166c007770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1664009830 tx=0x7f166402a5b0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.478+0000 7f167ad6a640 1 -- 172.21.15.117:0/4199998724 shutdown_connections 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.478+0000 7f167ad6a640 1 --2- 172.21.15.117:0/4199998724 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c0df7c0 0x7f166c007770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.478+0000 7f167ad6a640 1 --2- 172.21.15.117:0/4199998724 >> v2:172.21.15.117:3300/0 conn(0x7f166c049160 0x7f166c0d8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.479+0000 7f167ad6a640 1 -- 172.21.15.117:0/4199998724 >> 172.21.15.117:0/4199998724 conn(0x7f166c050fa0 msgr2=0x7f166c0513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.479+0000 7f167ad6a640 1 -- 172.21.15.117:0/4199998724 shutdown_connections 2024-09-17T05:35:55.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.479+0000 7f167ad6a640 1 -- 172.21.15.117:0/4199998724 wait complete. 2024-09-17T05:35:55.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.481+0000 7f167ad6a640 1 Processor -- start 2024-09-17T05:35:55.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.481+0000 7f167ad6a640 1 -- start start 2024-09-17T05:35:55.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.483+0000 7f167ad6a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f166c008d60 0x7f166c031a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:55.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.483+0000 7f1679d68640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f166c008d60 0x7f166c031a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.483+0000 7f1679d68640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f166c008d60 0x7f166c031a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:57234/0 (socket says 172.21.15.117:57234) 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.483+0000 7f1679d68640 1 -- 172.21.15.117:0/95191828 learned_addr learned my addr 172.21.15.117:0/95191828 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.483+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c049160 0x7f166c0d1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.484+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f166c0df7c0 0x7f166c04c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.484+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f166c007cb0 con 0x7f166c008d60 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.484+0000 7f1679567640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c049160 0x7f166c0d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.484+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f166c048960 con 0x7f166c0df7c0 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.484+0000 7f167a569640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f166c0df7c0 0x7f166c04c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.484+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f166c0dfbd0 con 0x7f166c049160 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f1679567640 1 -- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f166c0df7c0 msgr2=0x7f166c04c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f1679567640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f166c0df7c0 0x7f166c04c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f1679567640 1 -- 172.21.15.117:0/95191828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f166c008d60 msgr2=0x7f166c031a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f1679567640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f166c008d60 0x7f166c031a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f1679567640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c030b80 con 0x7f166c049160 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f1679567640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c049160 0x7f166c0d1870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1664031000 tx=0x7f1664033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664031d60 con 0x7f166c049160 2024-09-17T05:35:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c0d3f20 con 0x7f166c049160 2024-09-17T05:35:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.485+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1664043070 con 0x7f166c049160 2024-09-17T05:35:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.486+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664033330 con 0x7f166c049160 2024-09-17T05:35:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.486+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c0d1db0 con 0x7f166c049160 2024-09-17T05:35:55.488 INFO:tasks.ceph.mds.f.smithi119.stdout:starting mds.f at 2024-09-17T05:35:55.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.488+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c009170 con 0x7f166c049160 2024-09-17T05:35:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.488+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f16640334d0 con 0x7f166c049160 2024-09-17T05:35:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.488+0000 7f166a7fc640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1658078b90 0x7f165807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:35:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.489+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f16640bbf40 con 0x7f166c049160 2024-09-17T05:35:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.489+0000 7f1679d68640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1658078b90 0x7f165807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:35:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.490+0000 7f1679d68640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1658078b90 0x7f165807b0a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f167000bc00 tx=0x7f167000b390 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:35:55.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.498+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f166c0d3f20 con 0x7f166c049160 2024-09-17T05:35:55.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.925+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f166c050b80 con 0x7f166c049160 2024-09-17T05:35:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.926+0000 7f166a7fc640 1 -- 172.21.15.117:0/95191828 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6894 (secure 0 0 0) 0x7f1664031770 con 0x7f166c049160 2024-09-17T05:35:55.927 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:35:55.927 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":44,"btime":"2024-09-17T05:35:55:610284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5141,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/714171389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":714171389},{"type":"v1","addr":"172.21.15.119:6839","nonce":714171389}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:45.569656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:active","state_seq":20,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5457,"qdb_cluster":[5457]},"id":1}]} 2024-09-17T05:35:55.928 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 44 2024-09-17T05:35:55.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1658078b90 msgr2=0x7f165807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1658078b90 0x7f165807b0a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f167000bc00 tx=0x7f167000b390 comp rx=0 tx=0).stop 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c049160 msgr2=0x7f166c0d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c049160 0x7f166c0d1870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1664031000 tx=0x7f1664033ba0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 shutdown_connections 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f1658078b90 0x7f165807b0a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f166c0df7c0 0x7f166c04c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f166c049160 0x7f166c0d1870 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 --2- 172.21.15.117:0/95191828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f166c008d60 0x7f166c031a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.930+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 >> 172.21.15.117:0/95191828 conn(0x7f166c050fa0 msgr2=0x7f166c04ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.931+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 shutdown_connections 2024-09-17T05:35:55.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:35:55.931+0000 7f167ad6a640 1 -- 172.21.15.117:0/95191828 wait complete. 2024-09-17T05:35:55.951 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-09-17T05:35:55.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout:Syncing()...42 sec 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout:FFSB benchmark finished at: Tue Sep 17 05:36:23 2024 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout:Results: 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout:Benchmark took 382.09 sec 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout:Total Results 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout:=============== 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-17T05:36:23.794 INFO:tasks.workunit.client.0.smithi117.stdout: ======= ============ ========= ======= =========== ========== 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout: write : 6959360 18213.99 100.000% 100.000% 71.1MB/sec 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout:- 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout:18213.99 Transactions per Second 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout:Throughput Results 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout:=================== 2024-09-17T05:36:23.795 INFO:tasks.workunit.client.0.smithi117.stdout:Write Throughput: 71.1MB/sec 2024-09-17T05:36:23.826 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.826 INFO:tasks.workunit.client.0.smithi117.stdout:System Call Latency statistics in millisecs 2024-09-17T05:36:23.826 INFO:tasks.workunit.client.0.smithi117.stdout:===== 2024-09-17T05:36:23.826 INFO:tasks.workunit.client.0.smithi117.stdout: Min Avg Max Total Calls 2024-09-17T05:36:23.826 INFO:tasks.workunit.client.0.smithi117.stdout: ======== ======== ======== ============ 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout:[ open] 0.005000 0.269769 44.169998 5437 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[7] 20.000000 - 50.000000 : 8 2024-09-17T05:36:23.827 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[8] 10.000000 - 20.000000 : 24 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[9] 5.000000 - 10.000000 : 50 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[10] 2.000000 - 5.000000 : 111 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[11] 1.000000 - 2.000000 : 37 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[12] 0.500000 - 1.000000 : 10 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[13] 0.200000 - 0.500000 : 5 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[14] 0.100000 - 0.200000 : 17 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[15] 0.050000 - 0.100000 : 23 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[16] 0.020000 - 0.050000 : 881 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[17] 0.010000 - 0.020000 : 3613 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[18] 0.000000 - 0.010000 : 658 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.828 INFO:tasks.workunit.client.0.smithi117.stdout:[ write] 0.001000 1.436258 26954.599609 6959360 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[0] 5000.000000 - 10000.000000 : 315 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[1] 2000.000000 - 5000.000000 : 328 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[2] 1000.000000 - 2000.000000 : 173 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[3] 500.000000 - 1000.000000 : 254 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[4] 200.000000 - 500.000000 : 2766 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[5] 100.000000 - 200.000000 : 13747 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[6] 50.000000 - 100.000000 : 5521 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[7] 20.000000 - 50.000000 : 1506 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[8] 10.000000 - 20.000000 : 500 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[9] 5.000000 - 10.000000 : 559 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[10] 2.000000 - 5.000000 : 1078 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[11] 1.000000 - 2.000000 : 720 2024-09-17T05:36:23.829 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[12] 0.500000 - 1.000000 : 604 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[13] 0.200000 - 0.500000 : 840 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[14] 0.100000 - 0.200000 : 894 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[15] 0.050000 - 0.100000 : 1929 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[16] 0.020000 - 0.050000 : 10918 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[17] 0.010000 - 0.020000 : 47707 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[18] 0.000000 - 0.010000 : 6868793 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout:[ lseek] 0.000000 0.000646 55.476002 6959360 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:36:23.830 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[6] 50.000000 - 100.000000 : 3 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[7] 20.000000 - 50.000000 : 26 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[8] 10.000000 - 20.000000 : 29 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[9] 5.000000 - 10.000000 : 23 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[10] 2.000000 - 5.000000 : 42 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[11] 1.000000 - 2.000000 : 29 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[12] 0.500000 - 1.000000 : 41 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[13] 0.200000 - 0.500000 : 63 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[14] 0.100000 - 0.200000 : 61 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[15] 0.050000 - 0.100000 : 97 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[16] 0.020000 - 0.050000 : 509 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[17] 0.010000 - 0.020000 : 1463 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[18] 0.000000 - 0.010000 : 6956974 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.831 INFO:tasks.workunit.client.0.smithi117.stdout:[ close] 0.001000 0.004841 0.193000 5437 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-09-17T05:36:23.832 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[14] 0.100000 - 0.200000 : 2 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[15] 0.050000 - 0.100000 : 2 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[16] 0.020000 - 0.050000 : 9 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[17] 0.010000 - 0.020000 : 29 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: msec_range[18] 0.000000 - 0.010000 : 5395 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout:0.8% User Time 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout:6.5% System Time 2024-09-17T05:36:23.833 INFO:tasks.workunit.client.0.smithi117.stdout:7.4% CPU Utilization 2024-09-17T05:36:24.032 INFO:tasks.workunit.client.0.smithi117.stderr:+ cd .. 2024-09-17T05:36:24.033 INFO:tasks.workunit.client.0.smithi117.stderr:+ rm -r tmp ffsb 2024-09-17T05:36:26.660 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T05:36:26.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T05:36:26.767 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-09-17T05:36:26.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T05:36:27.423 DEBUG:teuthology.parallel:result is None 2024-09-17T05:36:27.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T05:36:27.520 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T05:36:27.520 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-09-17T05:36:27.531 INFO:tasks.mds_thrash:joining mds_thrasher 2024-09-17T05:36:27.531 ERROR:tasks.mds_thrash.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks/mds_thrash.py", line 121, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks/mds_thrash.py", line 237, in do_thrash self.sleep_unless_stopped(delay) File "/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks/thrasher.py", line 50, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-17T05:36:27.532 INFO:tasks.mds_thrash:done joining 2024-09-17T05:36:27.532 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-17T05:36:27.545 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-17T05:36:27.545 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:36:27.545 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:36:27.635 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-17T05:36:27.635 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:36:27.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:36:27.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.969+0000 7f2b6898f640 1 -- 172.21.15.117:0/1277660302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b64151cc0 msgr2=0x7f2b6415f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:27.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.969+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1277660302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b64151cc0 0x7f2b6415f080 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2b640c8cd0 tx=0x7f2b5402a530 comp rx=0 tx=0).stop 2024-09-17T05:36:27.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.972+0000 7f2b6898f640 1 -- 172.21.15.117:0/1277660302 shutdown_connections 2024-09-17T05:36:27.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.972+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1277660302 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b640c6e10 0x7f2b64165a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:27.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.972+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1277660302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b64151cc0 0x7f2b6415f080 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:27.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.972+0000 7f2b6898f640 1 -- 172.21.15.117:0/1277660302 >> 172.21.15.117:0/1277660302 conn(0x7f2b640cc460 msgr2=0x7f2b640cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:27.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.978+0000 7f2b6898f640 1 -- 172.21.15.117:0/1277660302 shutdown_connections 2024-09-17T05:36:27.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.981+0000 7f2b6898f640 1 -- 172.21.15.117:0/1277660302 wait complete. 2024-09-17T05:36:27.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.984+0000 7f2b6898f640 1 Processor -- start 2024-09-17T05:36:27.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.985+0000 7f2b6898f640 1 -- start start 2024-09-17T05:36:27.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.987+0000 7f2b6898f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 0x7f2b6413cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:27.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.988+0000 7f2b637fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 0x7f2b6413cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:27.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.988+0000 7f2b637fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 0x7f2b6413cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60440/0 (socket says 172.21.15.117:60440) 2024-09-17T05:36:27.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.988+0000 7f2b637fe640 1 -- 172.21.15.117:0/1224149328 learned_addr learned my addr 172.21.15.117:0/1224149328 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:27.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.988+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b640c88c0 0x7f2b640c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:27.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.988+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b64151cc0 0x7f2b640c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:27.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.989+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6415eb60 con 0x7f2b640c6e10 2024-09-17T05:36:27.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.989+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6415e880 con 0x7f2b640c88c0 2024-09-17T05:36:27.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.989+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f2b64073a20 con 0x7f2b64151cc0 2024-09-17T05:36:27.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.990+0000 7f2b63fff640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b64151cc0 0x7f2b640c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:27.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.991+0000 7f2b637fe640 1 -- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b64151cc0 msgr2=0x7f2b640c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:27.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.991+0000 7f2b637fe640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b64151cc0 0x7f2b640c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:27.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.991+0000 7f2b637fe640 1 -- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b640c88c0 msgr2=0x7f2b640c5cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:36:27.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.991+0000 7f2b637fe640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b640c88c0 0x7f2b640c5cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:27.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.991+0000 7f2b637fe640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6413d4d0 con 0x7f2b640c6e10 2024-09-17T05:36:27.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.992+0000 7f2b637fe640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 0x7f2b6413cf90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800b7b0 tx=0x7f2b5800bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:27.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.992+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58019070 con 0x7f2b640c6e10 2024-09-17T05:36:27.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.992+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b5800c810 con 0x7f2b640c6e10 2024-09-17T05:36:27.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.992+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b640c47b0 con 0x7f2b640c6e10 2024-09-17T05:36:27.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.994+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b64140b30 con 0x7f2b640c6e10 2024-09-17T05:36:27.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.994+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5800cc60 con 0x7f2b640c6e10 2024-09-17T05:36:27.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:27.996+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b64158a10 con 0x7f2b640c6e10 2024-09-17T05:36:28.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.003+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2b5801c070 con 0x7f2b640c6e10 2024-09-17T05:36:28.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.004+0000 7f2b4bfff640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b50078c60 0x7f2b5007b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:28.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.004+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f2b5809a510 con 0x7f2b640c6e10 2024-09-17T05:36:28.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.008+0000 7f2b62ffd640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b50078c60 0x7f2b5007b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:28.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.009+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2b58062690 con 0x7f2b640c6e10 2024-09-17T05:36:28.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.017+0000 7f2b62ffd640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b50078c60 0x7f2b5007b170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b54034000 tx=0x7f2b54004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.119:6839/2500745218 2024-09-18T05:35:25.910809+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.119:6838/2500745218 2024-09-18T05:35:25.910809+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.119:6835/2224210538 2024-09-18T05:33:15.875991+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.119:6834/2224210538 2024-09-18T05:33:15.875991+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6835/235108685 2024-09-18T05:31:45.857073+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6834/235108685 2024-09-18T05:31:45.857073+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6837/242246295 2024-09-18T05:29:10.830912+0000 2024-09-17T05:36:28.409 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6836/242246295 2024-09-18T05:29:10.830912+0000 2024-09-17T05:36:28.410 INFO:teuthology.orchestra.run.smithi117.stderr:listed 8 entries 2024-09-17T05:36:28.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.406+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2b64073800 con 0x7f2b640c6e10 2024-09-17T05:36:28.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.407+0000 7f2b4bfff640 1 -- 172.21.15.117:0/1224149328 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v31) ==== 80+0+492 (secure 0 0 0) 0x7f2b58061f90 con 0x7f2b640c6e10 2024-09-17T05:36:28.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.412+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b50078c60 msgr2=0x7f2b5007b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:28.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.412+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b50078c60 0x7f2b5007b170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b54034000 tx=0x7f2b54004730 comp rx=0 tx=0).stop 2024-09-17T05:36:28.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.412+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 msgr2=0x7f2b6413cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:28.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.412+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 0x7f2b6413cf90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800b7b0 tx=0x7f2b5800bc70 comp rx=0 tx=0).stop 2024-09-17T05:36:28.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.413+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 shutdown_connections 2024-09-17T05:36:28.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.413+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f2b50078c60 0x7f2b5007b170 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:28.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.413+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b64151cc0 0x7f2b640c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:28.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.413+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b640c88c0 0x7f2b640c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:28.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.413+0000 7f2b6898f640 1 --2- 172.21.15.117:0/1224149328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b640c6e10 0x7f2b6413cf90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:28.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.413+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 >> 172.21.15.117:0/1224149328 conn(0x7f2b640cc460 msgr2=0x7f2b6415b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:28.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.414+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 shutdown_connections 2024-09-17T05:36:28.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:28.414+0000 7f2b6898f640 1 -- 172.21.15.117:0/1224149328 wait complete. 2024-09-17T05:36:28.433 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T05:36:28.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:36:28.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T05:36:28.569 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:36:28.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:36:28.570 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T05:36:28.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:36:28.613 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-09-17T05:36:29.000 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T05:36:29.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:36:29.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T05:36:29.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:36:29.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link delete ceph-brx 2024-09-17T05:36:29.128 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T05:36:29.138 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T05:36:29.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:36:29.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.337+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/4127270595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 msgr2=0x7f5c180c94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.337+0000 7f5c1dcd5640 1 --2- 172.21.15.117:0/4127270595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 0x7f5c180c94a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c180bd8c0 tx=0x7f5c0802a4d0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.338+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/4127270595 shutdown_connections 2024-09-17T05:36:29.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.338+0000 7f5c1dcd5640 1 --2- 172.21.15.117:0/4127270595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 0x7f5c18151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.338+0000 7f5c1dcd5640 1 --2- 172.21.15.117:0/4127270595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 0x7f5c180c94a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.338+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/4127270595 >> 172.21.15.117:0/4127270595 conn(0x7f5c1806d190 msgr2=0x7f5c1806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:29.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.338+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/4127270595 shutdown_connections 2024-09-17T05:36:29.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.338+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/4127270595 wait complete. 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.341+0000 7f5c1dcd5640 1 Processor -- start 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.341+0000 7f5c1dcd5640 1 -- start start 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.343+0000 7f5c1dcd5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c180c7810 0x7f5c180cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.343+0000 7f5c1dcd5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 0x7f5c1814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.343+0000 7f5c1dcd5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 0x7f5c1815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.343+0000 7f5c177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c180c7810 0x7f5c180cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.344+0000 7f5c177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c180c7810 0x7f5c180cbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60456/0 (socket says 172.21.15.117:60456) 2024-09-17T05:36:29.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.344+0000 7f5c1dcd5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1815e920 con 0x7f5c180c7810 2024-09-17T05:36:29.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.344+0000 7f5c1dcd5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5c180c8340 con 0x7f5c1815bd10 2024-09-17T05:36:29.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.344+0000 7f5c1dcd5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5c18072e10 con 0x7f5c181518a0 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.344+0000 7f5c177fe640 1 -- 172.21.15.117:0/450651277 learned_addr learned my addr 172.21.15.117:0/450651277 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.346+0000 7f5c16ffd640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 0x7f5c1814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.347+0000 7f5c16ffd640 1 -- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 msgr2=0x7f5c1815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.347+0000 7f5c16ffd640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 0x7f5c1815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.347+0000 7f5c16ffd640 1 -- 172.21.15.117:0/450651277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c180c7810 msgr2=0x7f5c180cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.347+0000 7f5c16ffd640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c180c7810 0x7f5c180cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.347+0000 7f5c16ffd640 1 -- 172.21.15.117:0/450651277 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c180caef0 con 0x7f5c181518a0 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.347+0000 7f5c16ffd640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 0x7f5c1814e740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c08031000 tx=0x7f5c08033cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:29.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.348+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08031d60 con 0x7f5c181518a0 2024-09-17T05:36:29.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.348+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c08005020 con 0x7f5c181518a0 2024-09-17T05:36:29.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.349+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08031d60 con 0x7f5c181518a0 2024-09-17T05:36:29.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.351+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/450651277 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c180cc270 con 0x7f5c181518a0 2024-09-17T05:36:29.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.351+0000 7f5c1dcd5640 1 -- 172.21.15.117:0/450651277 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c180c6340 con 0x7f5c181518a0 2024-09-17T05:36:29.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.353+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c180cac60 con 0x7f5c181518a0 2024-09-17T05:36:29.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.354+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5c080051d0 con 0x7f5c181518a0 2024-09-17T05:36:29.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.355+0000 7f5bfffff640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5c04078fc0 0x7f5c0407b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.355+0000 7f5c177fe640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5c04078fc0 0x7f5c0407b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.356+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f5c08031b60 con 0x7f5c181518a0 2024-09-17T05:36:29.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.356+0000 7f5c177fe640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5c04078fc0 0x7f5c0407b4d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c006fd0 tx=0x7f5c0c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:29.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.366+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5c0803f030 con 0x7f5c181518a0 2024-09-17T05:36:29.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.688+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c1815de90 con 0x7f5c04078fc0 2024-09-17T05:36:29.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.700+0000 7f5bfffff640 1 -- 172.21.15.117:0/450651277 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+335532 (secure 0 0 0) 0x7f5c1815de90 con 0x7f5c04078fc0 2024-09-17T05:36:29.705 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:36:29.708 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5c04078fc0 msgr2=0x7f5c0407b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5c04078fc0 0x7f5c0407b4d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c006fd0 tx=0x7f5c0c008040 comp rx=0 tx=0).stop 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 msgr2=0x7f5c1814e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 0x7f5c1814e740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c08031000 tx=0x7f5c08033cc0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 shutdown_connections 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5c04078fc0 0x7f5c0407b4d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c1815bd10 0x7f5c1815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5c181518a0 0x7f5c1814e740 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 --2- 172.21.15.117:0/450651277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c180c7810 0x7f5c180cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 >> 172.21.15.117:0/450651277 conn(0x7f5c1806d190 msgr2=0x7f5c18151160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 shutdown_connections 2024-09-17T05:36:29.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.706+0000 7f5bfdffb640 1 -- 172.21.15.117:0/450651277 wait complete. 2024-09-17T05:36:29.726 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-17T05:36:29.528242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11887974460,"num_objects":3096,"num_object_clones":0,"num_object_copies":12202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":937,"num_read_kb":78335,"num_write":2621667,"num_write_kb":32805354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177852,"ondisk_log_size":177852,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":20143836,"kb_used_data":12571996,"kb_used_omap":12,"kb_used_meta":7571699,"kb_avail":734830884,"statfs":{"total":773094113280,"available":752466825216,"internally_reserved":0,"allocated":12873723904,"data_stored":24810464016,"data_compressed":1273328494,"data_compressed_allocated":5212246016,"data_compressed_original":17150644224,"omap_allocated":12751,"internal_metadata":7753420337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,2,2,6,5,5,0,0,30,93,11],"upper_bound":16384},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1617300524,"num_objects":-386,"num_object_clones":0,"num_object_copies":-1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-386,"num_whiteouts":0,"num_read":93,"num_read_kb":12,"num_write":38695,"num_write_kb":174965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003492"},"pg_stats":[{"pgid":"2.1f","version":"30'18","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:45.149722+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:35:45.149722+0000","last_peered":"2024-09-17T05:35:45.149722+0000","last_clean":"2024-09-17T05:35:45.149722+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:35:45.149722+0000","last_undegraded":"2024-09-17T05:35:45.149722+0000","last_fullsized":"2024-09-17T05:35:45.149722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":4124,"num_write":15,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970029+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:35:25.970029+0000","last_peered":"2024-09-17T05:35:25.970029+0000","last_clean":"2024-09-17T05:35:25.970029+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:35:25.970029+0000","last_undegraded":"2024-09-17T05:35:25.970029+0000","last_fullsized":"2024-09-17T05:35:25.970029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"31'5749","reported_seq":5796,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:23.369232+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:36:23.369232+0000","last_peered":"2024-09-17T05:36:23.369232+0000","last_clean":"2024-09-17T05:36:23.369232+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:36:23.369232+0000","last_undegraded":"2024-09-17T05:36:23.369232+0000","last_fullsized":"2024-09-17T05:36:23.369232+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440520717,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":86546,"num_write_kb":1098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"29'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.956723+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:35:30.956723+0000","last_peered":"2024-09-17T05:35:30.956723+0000","last_clean":"2024-09-17T05:35:30.956723+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:35:30.956723+0000","last_undegraded":"2024-09-17T05:35:30.956723+0000","last_fullsized":"2024-09-17T05:35:30.956723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952279+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:35:25.952279+0000","last_peered":"2024-09-17T05:35:25.952279+0000","last_clean":"2024-09-17T05:35:25.952279+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:35:25.952279+0000","last_undegraded":"2024-09-17T05:35:25.952279+0000","last_fullsized":"2024-09-17T05:35:25.952279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"31'5672","reported_seq":5720,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.170725+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:36:25.170725+0000","last_peered":"2024-09-17T05:36:25.170725+0000","last_clean":"2024-09-17T05:36:25.170725+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:36:25.170725+0000","last_undegraded":"2024-09-17T05:36:25.170725+0000","last_fullsized":"2024-09-17T05:36:25.170725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402678953,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":85001,"num_write_kb":1045061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"31'49","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.841015+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:35:50.841015+0000","last_peered":"2024-09-17T05:35:50.841015+0000","last_clean":"2024-09-17T05:35:50.841015+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:35:50.841015+0000","last_undegraded":"2024-09-17T05:35:50.841015+0000","last_fullsized":"2024-09-17T05:35:50.841015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":8212,"num_write":40,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.969942+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:35:25.969942+0000","last_peered":"2024-09-17T05:35:25.969942+0000","last_clean":"2024-09-17T05:35:25.969942+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:35:25.969942+0000","last_undegraded":"2024-09-17T05:35:25.969942+0000","last_fullsized":"2024-09-17T05:35:25.969942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'5296","reported_seq":5345,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.978235+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:36:24.978235+0000","last_peered":"2024-09-17T05:36:24.978235+0000","last_clean":"2024-09-17T05:36:24.978235+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:36:24.978235+0000","last_undegraded":"2024-09-17T05:36:24.978235+0000","last_fullsized":"2024-09-17T05:36:24.978235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398475913,"num_objects":105,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79368,"num_write_kb":982831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"31'198","reported_seq":239,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.752100+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:36:26.752100+0000","last_peered":"2024-09-17T05:36:26.752100+0000","last_clean":"2024-09-17T05:36:26.752100+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:36:26.752100+0000","last_undegraded":"2024-09-17T05:36:26.752100+0000","last_fullsized":"2024-09-17T05:36:26.752100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395069,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":83,"num_write":197,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065126+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:35:26.065126+0000","last_peered":"2024-09-17T05:35:26.065126+0000","last_clean":"2024-09-17T05:35:26.065126+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:35:26.065126+0000","last_undegraded":"2024-09-17T05:35:26.065126+0000","last_fullsized":"2024-09-17T05:35:26.065126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"31'4547","reported_seq":4594,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.247886+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:36:25.247886+0000","last_peered":"2024-09-17T05:36:25.247886+0000","last_clean":"2024-09-17T05:36:25.247886+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:36:25.247886+0000","last_undegraded":"2024-09-17T05:36:25.247886+0000","last_fullsized":"2024-09-17T05:36:25.247886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348151202,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":68171,"num_write_kb":860103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335717+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:35:26.335717+0000","last_peered":"2024-09-17T05:35:26.335717+0000","last_clean":"2024-09-17T05:35:26.335717+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:35:26.335717+0000","last_undegraded":"2024-09-17T05:35:26.335717+0000","last_fullsized":"2024-09-17T05:35:26.335717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"31'126","reported_seq":169,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.838914+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:36:19.838914+0000","last_peered":"2024-09-17T05:36:19.838914+0000","last_clean":"2024-09-17T05:36:19.838914+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:36:19.838914+0000","last_undegraded":"2024-09-17T05:36:19.838914+0000","last_fullsized":"2024-09-17T05:36:19.838914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4104,"num_write":119,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"31'5574","reported_seq":5615,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.846827+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:36:22.846827+0000","last_peered":"2024-09-17T05:36:22.846827+0000","last_clean":"2024-09-17T05:36:22.846827+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:36:22.846827+0000","last_undegraded":"2024-09-17T05:36:22.846827+0000","last_fullsized":"2024-09-17T05:36:22.846827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419575348,"num_objects":107,"num_object_clones":0,"num_object_copies":428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":83636,"num_write_kb":1023293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947658+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:35:25.947658+0000","last_peered":"2024-09-17T05:35:25.947658+0000","last_clean":"2024-09-17T05:35:25.947658+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:35:25.947658+0000","last_undegraded":"2024-09-17T05:35:25.947658+0000","last_fullsized":"2024-09-17T05:35:25.947658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"31'6","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.663343+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:35:34.663343+0000","last_peered":"2024-09-17T05:35:34.663343+0000","last_clean":"2024-09-17T05:35:34.663343+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:35:34.663343+0000","last_undegraded":"2024-09-17T05:35:34.663343+0000","last_fullsized":"2024-09-17T05:35:34.663343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"31'5484","reported_seq":5529,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.331686+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:36:28.331686+0000","last_peered":"2024-09-17T05:36:28.331686+0000","last_clean":"2024-09-17T05:36:28.331686+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:36:28.331686+0000","last_undegraded":"2024-09-17T05:36:28.331686+0000","last_fullsized":"2024-09-17T05:36:28.331686+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268458584,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":81905,"num_write_kb":1031804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.012651+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:35:26.012651+0000","last_peered":"2024-09-17T05:35:26.012651+0000","last_clean":"2024-09-17T05:35:26.012651+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:35:26.012651+0000","last_undegraded":"2024-09-17T05:35:26.012651+0000","last_fullsized":"2024-09-17T05:35:26.012651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"30'4","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953261+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:35:25.953261+0000","last_peered":"2024-09-17T05:35:25.953261+0000","last_clean":"2024-09-17T05:35:25.953261+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:35:25.953261+0000","last_undegraded":"2024-09-17T05:35:25.953261+0000","last_fullsized":"2024-09-17T05:35:25.953261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'6469","reported_seq":6509,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.260774+0000","last_change":"2024-09-17T05:28:07.483827+0000","last_active":"2024-09-17T05:36:28.260774+0000","last_peered":"2024-09-17T05:36:28.260774+0000","last_clean":"2024-09-17T05:36:28.260774+0000","last_became_active":"2024-09-17T05:28:07.483263+0000","last_became_peered":"2024-09-17T05:28:07.483263+0000","last_unstale":"2024-09-17T05:36:28.260774+0000","last_undegraded":"2024-09-17T05:36:28.260774+0000","last_fullsized":"2024-09-17T05:36:28.260774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.655059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306281587,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":96582,"num_write_kb":1196264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2,3],"acting":[0,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.074006+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:35:26.074006+0000","last_peered":"2024-09-17T05:35:26.074006+0000","last_clean":"2024-09-17T05:35:26.074006+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:35:26.074006+0000","last_undegraded":"2024-09-17T05:35:26.074006+0000","last_fullsized":"2024-09-17T05:35:26.074006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"30'13","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.893052+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:35:31.893052+0000","last_peered":"2024-09-17T05:35:31.893052+0000","last_clean":"2024-09-17T05:35:31.893052+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:35:31.893052+0000","last_undegraded":"2024-09-17T05:35:31.893052+0000","last_fullsized":"2024-09-17T05:35:31.893052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":12,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"31'4761","reported_seq":4805,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.831627+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:36:24.831627+0000","last_peered":"2024-09-17T05:36:24.831627+0000","last_clean":"2024-09-17T05:36:24.831627+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:36:24.831627+0000","last_undegraded":"2024-09-17T05:36:24.831627+0000","last_fullsized":"2024-09-17T05:36:24.831627+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360925861,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":71329,"num_write_kb":889518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"28'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.940299+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:35:25.940299+0000","last_peered":"2024-09-17T05:35:25.940299+0000","last_clean":"2024-09-17T05:35:25.940299+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:35:25.940299+0000","last_undegraded":"2024-09-17T05:35:25.940299+0000","last_fullsized":"2024-09-17T05:35:25.940299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:04.068629+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:36:04.068629+0000","last_peered":"2024-09-17T05:36:04.068629+0000","last_clean":"2024-09-17T05:36:04.068629+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:36:04.068629+0000","last_undegraded":"2024-09-17T05:36:04.068629+0000","last_fullsized":"2024-09-17T05:36:04.068629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"31'5448","reported_seq":5497,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.113608+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:36:25.113608+0000","last_peered":"2024-09-17T05:36:25.113608+0000","last_clean":"2024-09-17T05:36:25.113608+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:36:25.113608+0000","last_undegraded":"2024-09-17T05:36:25.113608+0000","last_fullsized":"2024-09-17T05:36:25.113608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402664508,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":81667,"num_write_kb":1039133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"28'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.961318+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:35:25.961318+0000","last_peered":"2024-09-17T05:35:25.961318+0000","last_clean":"2024-09-17T05:35:25.961318+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:35:25.961318+0000","last_undegraded":"2024-09-17T05:35:25.961318+0000","last_fullsized":"2024-09-17T05:35:25.961318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"31'214","reported_seq":258,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.682203+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:36:24.682203+0000","last_peered":"2024-09-17T05:36:24.682203+0000","last_clean":"2024-09-17T05:36:24.682203+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:36:24.682203+0000","last_undegraded":"2024-09-17T05:36:24.682203+0000","last_fullsized":"2024-09-17T05:36:24.682203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":21,"num_write":434,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"31'5919","reported_seq":5970,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.298032+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:36:25.298032+0000","last_peered":"2024-09-17T05:36:25.298032+0000","last_clean":"2024-09-17T05:36:25.298032+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:36:25.298032+0000","last_undegraded":"2024-09-17T05:36:25.298032+0000","last_fullsized":"2024-09-17T05:36:25.298032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5919,"log_dups_size":0,"ondisk_log_size":5919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411056495,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":88791,"num_write_kb":1123429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.573647+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:35:55.573647+0000","last_peered":"2024-09-17T05:35:55.573647+0000","last_clean":"2024-09-17T05:35:55.573647+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:35:55.573647+0000","last_undegraded":"2024-09-17T05:35:55.573647+0000","last_fullsized":"2024-09-17T05:35:55.573647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.066159+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:35:26.066159+0000","last_peered":"2024-09-17T05:35:26.066159+0000","last_clean":"2024-09-17T05:35:26.066159+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:35:26.066159+0000","last_undegraded":"2024-09-17T05:35:26.066159+0000","last_fullsized":"2024-09-17T05:35:26.066159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'5008","reported_seq":5050,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.326050+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:36:27.326050+0000","last_peered":"2024-09-17T05:36:27.326050+0000","last_clean":"2024-09-17T05:36:27.326050+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:36:27.326050+0000","last_undegraded":"2024-09-17T05:36:27.326050+0000","last_fullsized":"2024-09-17T05:36:27.326050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276833139,"num_objects":70,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":74950,"num_write_kb":939196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"31'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:59.340987+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:35:59.340987+0000","last_peered":"2024-09-17T05:35:59.340987+0000","last_clean":"2024-09-17T05:35:59.340987+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:35:59.340987+0000","last_undegraded":"2024-09-17T05:35:59.340987+0000","last_fullsized":"2024-09-17T05:35:59.340987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'1","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968299+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:35:25.968299+0000","last_peered":"2024-09-17T05:35:25.968299+0000","last_clean":"2024-09-17T05:35:25.968299+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:35:25.968299+0000","last_undegraded":"2024-09-17T05:35:25.968299+0000","last_fullsized":"2024-09-17T05:35:25.968299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"31'6130","reported_seq":6178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.692134+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:36:27.692134+0000","last_peered":"2024-09-17T05:36:27.692134+0000","last_clean":"2024-09-17T05:36:27.692134+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:36:27.692134+0000","last_undegraded":"2024-09-17T05:36:27.692134+0000","last_fullsized":"2024-09-17T05:36:27.692134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381700523,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":91961,"num_write_kb":1145505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.950313+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:36:27.950313+0000","last_peered":"2024-09-17T05:36:27.950313+0000","last_clean":"2024-09-17T05:36:27.950313+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:36:27.950313+0000","last_undegraded":"2024-09-17T05:36:27.950313+0000","last_fullsized":"2024-09-17T05:36:27.950313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"30'41","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.332894+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:35:32.332894+0000","last_peered":"2024-09-17T05:35:32.332894+0000","last_clean":"2024-09-17T05:35:32.332894+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:35:32.332894+0000","last_undegraded":"2024-09-17T05:35:32.332894+0000","last_fullsized":"2024-09-17T05:35:32.332894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":35,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'5280","reported_seq":5330,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.478014+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:36:27.478014+0000","last_peered":"2024-09-17T05:36:27.478014+0000","last_clean":"2024-09-17T05:36:27.478014+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:36:27.478014+0000","last_undegraded":"2024-09-17T05:36:27.478014+0000","last_fullsized":"2024-09-17T05:36:27.478014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310455443,"num_objects":80,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":78953,"num_write_kb":985615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949408+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:35:25.949408+0000","last_peered":"2024-09-17T05:35:25.949408+0000","last_clean":"2024-09-17T05:35:25.949408+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:35:25.949408+0000","last_undegraded":"2024-09-17T05:35:25.949408+0000","last_fullsized":"2024-09-17T05:35:25.949408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"31'27","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.695352+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:36:24.695352+0000","last_peered":"2024-09-17T05:36:24.695352+0000","last_clean":"2024-09-17T05:36:24.695352+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:36:24.695352+0000","last_undegraded":"2024-09-17T05:36:24.695352+0000","last_fullsized":"2024-09-17T05:36:24.695352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61980,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":83,"num_write":24,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'5837","reported_seq":5888,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.309862+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:36:27.309862+0000","last_peered":"2024-09-17T05:36:27.309862+0000","last_clean":"2024-09-17T05:36:27.309862+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:36:27.309862+0000","last_undegraded":"2024-09-17T05:36:27.309862+0000","last_fullsized":"2024-09-17T05:36:27.309862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356569031,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":87303,"num_write_kb":1093347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962251+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:35:25.962251+0000","last_peered":"2024-09-17T05:35:25.962251+0000","last_clean":"2024-09-17T05:35:25.962251+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:35:25.962251+0000","last_undegraded":"2024-09-17T05:35:25.962251+0000","last_fullsized":"2024-09-17T05:35:25.962251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"30'91","reported_seq":136,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.303333+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:35:31.303333+0000","last_peered":"2024-09-17T05:35:31.303333+0000","last_clean":"2024-09-17T05:35:31.303333+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:35:31.303333+0000","last_undegraded":"2024-09-17T05:35:31.303333+0000","last_fullsized":"2024-09-17T05:35:31.303333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8204,"num_write":88,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"31'4749","reported_seq":4791,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.976651+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:36:26.976651+0000","last_peered":"2024-09-17T05:36:26.976651+0000","last_clean":"2024-09-17T05:36:26.976651+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:36:26.976651+0000","last_undegraded":"2024-09-17T05:36:26.976651+0000","last_fullsized":"2024-09-17T05:36:26.976651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4749,"log_dups_size":0,"ondisk_log_size":4749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294034242,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":70943,"num_write_kb":889582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.940713+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:35:25.940713+0000","last_peered":"2024-09-17T05:35:25.940713+0000","last_clean":"2024-09-17T05:35:25.940713+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:35:25.940713+0000","last_undegraded":"2024-09-17T05:35:25.940713+0000","last_fullsized":"2024-09-17T05:35:25.940713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.201104+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:35:26.201104+0000","last_peered":"2024-09-17T05:35:26.201104+0000","last_clean":"2024-09-17T05:35:26.201104+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:35:26.201104+0000","last_undegraded":"2024-09-17T05:35:26.201104+0000","last_fullsized":"2024-09-17T05:35:26.201104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":135,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187694+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:35:26.187694+0000","last_peered":"2024-09-17T05:35:26.187694+0000","last_clean":"2024-09-17T05:35:26.187694+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:35:26.187694+0000","last_undegraded":"2024-09-17T05:35:26.187694+0000","last_fullsized":"2024-09-17T05:35:26.187694+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'5318","reported_seq":5367,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.734064+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:36:24.734064+0000","last_peered":"2024-09-17T05:36:24.734064+0000","last_clean":"2024-09-17T05:36:24.734064+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:36:24.734064+0000","last_undegraded":"2024-09-17T05:36:24.734064+0000","last_fullsized":"2024-09-17T05:36:24.734064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398468612,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":79949,"num_write_kb":1015697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065270+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:35:26.065270+0000","last_peered":"2024-09-17T05:35:26.065270+0000","last_clean":"2024-09-17T05:35:26.065270+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:35:26.065270+0000","last_undegraded":"2024-09-17T05:35:26.065270+0000","last_fullsized":"2024-09-17T05:35:26.065270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"30'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.981638+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:35:25.981638+0000","last_peered":"2024-09-17T05:35:25.981638+0000","last_clean":"2024-09-17T05:35:25.981638+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:35:25.981638+0000","last_undegraded":"2024-09-17T05:35:25.981638+0000","last_fullsized":"2024-09-17T05:35:25.981638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"31'5485","reported_seq":5533,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.259874+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:36:25.259874+0000","last_peered":"2024-09-17T05:36:25.259874+0000","last_clean":"2024-09-17T05:36:25.259874+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:36:25.259874+0000","last_undegraded":"2024-09-17T05:36:25.259874+0000","last_fullsized":"2024-09-17T05:36:25.259874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373295017,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82225,"num_write_kb":1025442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943229+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:35:25.943229+0000","last_peered":"2024-09-17T05:35:25.943229+0000","last_clean":"2024-09-17T05:35:25.943229+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:35:25.943229+0000","last_undegraded":"2024-09-17T05:35:25.943229+0000","last_fullsized":"2024-09-17T05:35:25.943229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"30'20","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.072706+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:35:32.072706+0000","last_peered":"2024-09-17T05:35:32.072706+0000","last_clean":"2024-09-17T05:35:32.072706+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:35:32.072706+0000","last_undegraded":"2024-09-17T05:35:32.072706+0000","last_fullsized":"2024-09-17T05:35:32.072706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":19,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"31'5002","reported_seq":5050,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.107012+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:36:27.107012+0000","last_peered":"2024-09-17T05:36:27.107012+0000","last_clean":"2024-09-17T05:36:27.107012+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:36:27.107012+0000","last_undegraded":"2024-09-17T05:36:27.107012+0000","last_fullsized":"2024-09-17T05:36:27.107012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314598708,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":74882,"num_write_kb":933970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.946846+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:35:25.946846+0000","last_peered":"2024-09-17T05:35:25.946846+0000","last_clean":"2024-09-17T05:35:25.946846+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:35:25.946846+0000","last_undegraded":"2024-09-17T05:35:25.946846+0000","last_fullsized":"2024-09-17T05:35:25.946846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'27","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.808293+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:35:50.808293+0000","last_peered":"2024-09-17T05:35:50.808293+0000","last_clean":"2024-09-17T05:35:50.808293+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:35:50.808293+0000","last_undegraded":"2024-09-17T05:35:50.808293+0000","last_fullsized":"2024-09-17T05:35:50.808293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":26,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"31'5083","reported_seq":5128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:20.292547+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:36:20.292547+0000","last_peered":"2024-09-17T05:36:20.292547+0000","last_clean":"2024-09-17T05:36:20.292547+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:36:20.292547+0000","last_undegraded":"2024-09-17T05:36:20.292547+0000","last_fullsized":"2024-09-17T05:36:20.292547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394284666,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76376,"num_write_kb":955028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936873+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:35:25.936873+0000","last_peered":"2024-09-17T05:35:25.936873+0000","last_clean":"2024-09-17T05:35:25.936873+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:35:25.936873+0000","last_undegraded":"2024-09-17T05:35:25.936873+0000","last_fullsized":"2024-09-17T05:35:25.936873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"31'8","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.592840+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:35:34.592840+0000","last_peered":"2024-09-17T05:35:34.592840+0000","last_clean":"2024-09-17T05:35:34.592840+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:35:34.592840+0000","last_undegraded":"2024-09-17T05:35:34.592840+0000","last_fullsized":"2024-09-17T05:35:34.592840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"31'5113","reported_seq":5158,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.336370+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:36:26.336370+0000","last_peered":"2024-09-17T05:36:26.336370+0000","last_clean":"2024-09-17T05:36:26.336370+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:36:26.336370+0000","last_undegraded":"2024-09-17T05:36:26.336370+0000","last_fullsized":"2024-09-17T05:36:26.336370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344012237,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76601,"num_write_kb":970760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.073783+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:35:26.073783+0000","last_peered":"2024-09-17T05:35:26.073783+0000","last_clean":"2024-09-17T05:35:26.073783+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:35:26.073783+0000","last_undegraded":"2024-09-17T05:35:26.073783+0000","last_fullsized":"2024-09-17T05:35:26.073783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"30'19","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.833437+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:35:50.833437+0000","last_peered":"2024-09-17T05:35:50.833437+0000","last_clean":"2024-09-17T05:35:50.833437+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:35:50.833437+0000","last_undegraded":"2024-09-17T05:35:50.833437+0000","last_fullsized":"2024-09-17T05:35:50.833437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":40,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"31'5738","reported_seq":5789,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.859102+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:36:26.859102+0000","last_peered":"2024-09-17T05:36:26.859102+0000","last_clean":"2024-09-17T05:36:26.859102+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:36:26.859102+0000","last_undegraded":"2024-09-17T05:36:26.859102+0000","last_fullsized":"2024-09-17T05:36:26.859102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331603745,"num_objects":90,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":85828,"num_write_kb":1086559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187433+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:35:26.187433+0000","last_peered":"2024-09-17T05:35:26.187433+0000","last_clean":"2024-09-17T05:35:26.187433+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:35:26.187433+0000","last_undegraded":"2024-09-17T05:35:26.187433+0000","last_fullsized":"2024-09-17T05:35:26.187433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"30'13","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943104+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:35:25.943104+0000","last_peered":"2024-09-17T05:35:25.943104+0000","last_clean":"2024-09-17T05:35:25.943104+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:35:25.943104+0000","last_undegraded":"2024-09-17T05:35:25.943104+0000","last_fullsized":"2024-09-17T05:35:25.943104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1689,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":13,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"31'5839","reported_seq":5887,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.046398+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:36:27.046398+0000","last_peered":"2024-09-17T05:36:27.046398+0000","last_clean":"2024-09-17T05:36:27.046398+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:36:27.046398+0000","last_undegraded":"2024-09-17T05:36:27.046398+0000","last_fullsized":"2024-09-17T05:36:27.046398+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339798246,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87347,"num_write_kb":1070021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.200625+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:35:26.200625+0000","last_peered":"2024-09-17T05:35:26.200625+0000","last_clean":"2024-09-17T05:35:26.200625+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:35:26.200625+0000","last_undegraded":"2024-09-17T05:35:26.200625+0000","last_fullsized":"2024-09-17T05:35:26.200625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"30'6","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335841+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:35:26.335841+0000","last_peered":"2024-09-17T05:35:26.335841+0000","last_clean":"2024-09-17T05:35:26.335841+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:35:26.335841+0000","last_undegraded":"2024-09-17T05:35:26.335841+0000","last_fullsized":"2024-09-17T05:35:26.335841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'5863","reported_seq":5910,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.484944+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:36:27.484944+0000","last_peered":"2024-09-17T05:36:27.484944+0000","last_clean":"2024-09-17T05:36:27.484944+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:36:27.484944+0000","last_undegraded":"2024-09-17T05:36:27.484944+0000","last_fullsized":"2024-09-17T05:36:27.484944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5863,"log_dups_size":0,"ondisk_log_size":5863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318999243,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87526,"num_write_kb":1085839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"28'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970702+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:35:25.970702+0000","last_peered":"2024-09-17T05:35:25.970702+0000","last_clean":"2024-09-17T05:35:25.970702+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:35:25.970702+0000","last_undegraded":"2024-09-17T05:35:25.970702+0000","last_fullsized":"2024-09-17T05:35:25.970702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"30'87","reported_seq":134,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.501205+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:35:31.501205+0000","last_peered":"2024-09-17T05:35:31.501205+0000","last_clean":"2024-09-17T05:35:31.501205+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:35:31.501205+0000","last_undegraded":"2024-09-17T05:35:31.501205+0000","last_fullsized":"2024-09-17T05:35:31.501205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":8218,"num_write":88,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"31'5132","reported_seq":5178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.833860+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:36:22.833860+0000","last_peered":"2024-09-17T05:36:22.833860+0000","last_clean":"2024-09-17T05:36:22.833860+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:36:22.833860+0000","last_undegraded":"2024-09-17T05:36:22.833860+0000","last_fullsized":"2024-09-17T05:36:22.833860+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394264576,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":76895,"num_write_kb":960359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.945356+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:35:25.945356+0000","last_peered":"2024-09-17T05:35:25.945356+0000","last_clean":"2024-09-17T05:35:25.945356+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:35:25.945356+0000","last_undegraded":"2024-09-17T05:35:25.945356+0000","last_fullsized":"2024-09-17T05:35:25.945356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"31'213","reported_seq":258,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:15.763813+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:36:15.763813+0000","last_peered":"2024-09-17T05:36:15.763813+0000","last_clean":"2024-09-17T05:36:15.763813+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:36:15.763813+0000","last_undegraded":"2024-09-17T05:36:15.763813+0000","last_fullsized":"2024-09-17T05:36:15.763813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":212,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"31'5323","reported_seq":5372,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.196028+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:36:25.196028+0000","last_peered":"2024-09-17T05:36:25.196028+0000","last_clean":"2024-09-17T05:36:25.196028+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:36:25.196028+0000","last_undegraded":"2024-09-17T05:36:25.196028+0000","last_fullsized":"2024-09-17T05:36:25.196028+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390153058,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79495,"num_write_kb":1008039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997367+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:35:25.997367+0000","last_peered":"2024-09-17T05:35:25.997367+0000","last_clean":"2024-09-17T05:35:25.997367+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:35:25.997367+0000","last_undegraded":"2024-09-17T05:35:25.997367+0000","last_fullsized":"2024-09-17T05:35:25.997367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"31'10","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.839506+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:35:50.839506+0000","last_peered":"2024-09-17T05:35:50.839506+0000","last_clean":"2024-09-17T05:35:50.839506+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:35:50.839506+0000","last_undegraded":"2024-09-17T05:35:50.839506+0000","last_fullsized":"2024-09-17T05:35:50.839506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"31'5584","reported_seq":5628,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.389841+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:36:25.389841+0000","last_peered":"2024-09-17T05:36:25.389841+0000","last_clean":"2024-09-17T05:36:25.389841+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:36:25.389841+0000","last_undegraded":"2024-09-17T05:36:25.389841+0000","last_fullsized":"2024-09-17T05:36:25.389841+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390177777,"num_objects":103,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":15,"num_read_kb":0,"num_write":83378,"num_write_kb":1022097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952372+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:35:25.952372+0000","last_peered":"2024-09-17T05:35:25.952372+0000","last_clean":"2024-09-17T05:35:25.952372+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:35:25.952372+0000","last_undegraded":"2024-09-17T05:35:25.952372+0000","last_fullsized":"2024-09-17T05:35:25.952372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"30'3","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943930+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:35:25.943930+0000","last_peered":"2024-09-17T05:35:25.943930+0000","last_clean":"2024-09-17T05:35:25.943930+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:35:25.943930+0000","last_undegraded":"2024-09-17T05:35:25.943930+0000","last_fullsized":"2024-09-17T05:35:25.943930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"31'5306","reported_seq":5349,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.902604+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:36:24.902604+0000","last_peered":"2024-09-17T05:36:24.902604+0000","last_clean":"2024-09-17T05:36:24.902604+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:36:24.902604+0000","last_undegraded":"2024-09-17T05:36:24.902604+0000","last_fullsized":"2024-09-17T05:36:24.902604+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385944448,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":13,"num_read_kb":0,"num_write":79626,"num_write_kb":1002963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936737+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:35:25.936737+0000","last_peered":"2024-09-17T05:35:25.936737+0000","last_clean":"2024-09-17T05:35:25.936737+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:35:25.936737+0000","last_undegraded":"2024-09-17T05:35:25.936737+0000","last_fullsized":"2024-09-17T05:35:25.936737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"31'11","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.972204+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:35:25.972204+0000","last_peered":"2024-09-17T05:35:25.972204+0000","last_clean":"2024-09-17T05:35:25.972204+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:35:25.972204+0000","last_undegraded":"2024-09-17T05:35:25.972204+0000","last_fullsized":"2024-09-17T05:35:25.972204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'5506","reported_seq":5553,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.983518+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:36:24.983518+0000","last_peered":"2024-09-17T05:36:24.983518+0000","last_clean":"2024-09-17T05:36:24.983518+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:36:24.983518+0000","last_undegraded":"2024-09-17T05:36:24.983518+0000","last_fullsized":"2024-09-17T05:36:24.983518+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406877841,"num_objects":103,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82610,"num_write_kb":1020969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6379","reported_seq":6427,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.313591+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:36:25.313591+0000","last_peered":"2024-09-17T05:36:25.313591+0000","last_clean":"2024-09-17T05:36:25.313591+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:36:25.313591+0000","last_undegraded":"2024-09-17T05:36:25.313591+0000","last_fullsized":"2024-09-17T05:36:25.313591+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452996234,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":95842,"num_write_kb":1202164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953061+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:35:25.953061+0000","last_peered":"2024-09-17T05:35:25.953061+0000","last_clean":"2024-09-17T05:35:25.953061+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:35:25.953061+0000","last_undegraded":"2024-09-17T05:35:25.953061+0000","last_fullsized":"2024-09-17T05:35:25.953061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"30'27","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.703337+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:35:31.703337+0000","last_peered":"2024-09-17T05:35:31.703337+0000","last_clean":"2024-09-17T05:35:31.703337+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:35:31.703337+0000","last_undegraded":"2024-09-17T05:35:31.703337+0000","last_fullsized":"2024-09-17T05:35:31.703337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":24,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"31'5525","reported_seq":5577,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:23.122693+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:36:23.122693+0000","last_peered":"2024-09-17T05:36:23.122693+0000","last_clean":"2024-09-17T05:36:23.122693+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:36:23.122693+0000","last_undegraded":"2024-09-17T05:36:23.122693+0000","last_fullsized":"2024-09-17T05:36:23.122693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427902156,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":82949,"num_write_kb":1044536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"28'14","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949598+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:35:25.949598+0000","last_peered":"2024-09-17T05:35:25.949598+0000","last_clean":"2024-09-17T05:35:25.949598+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:35:25.949598+0000","last_undegraded":"2024-09-17T05:35:25.949598+0000","last_fullsized":"2024-09-17T05:35:25.949598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"30'5","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962325+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:35:25.962325+0000","last_peered":"2024-09-17T05:35:25.962325+0000","last_clean":"2024-09-17T05:35:25.962325+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:35:25.962325+0000","last_undegraded":"2024-09-17T05:35:25.962325+0000","last_fullsized":"2024-09-17T05:35:25.962325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"31'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.941350+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:35:30.941350+0000","last_peered":"2024-09-17T05:35:30.941350+0000","last_clean":"2024-09-17T05:35:30.941350+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:35:30.941350+0000","last_undegraded":"2024-09-17T05:35:30.941350+0000","last_fullsized":"2024-09-17T05:35:30.941350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4108,"num_write":17,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941766+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:35:25.941766+0000","last_peered":"2024-09-17T05:35:25.941766+0000","last_clean":"2024-09-17T05:35:25.941766+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:35:25.941766+0000","last_undegraded":"2024-09-17T05:35:25.941766+0000","last_fullsized":"2024-09-17T05:35:25.941766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4537","reported_seq":4575,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.435603+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:36:25.435603+0000","last_peered":"2024-09-17T05:36:25.435603+0000","last_clean":"2024-09-17T05:36:25.435603+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:36:25.435603+0000","last_undegraded":"2024-09-17T05:36:25.435603+0000","last_fullsized":"2024-09-17T05:36:25.435603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297797311,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":68041,"num_write_kb":833128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1234","reported_seq":1275,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.957097+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:35:25.957097+0000","last_peered":"2024-09-17T05:35:25.957097+0000","last_clean":"2024-09-17T05:35:25.957097+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:35:25.957097+0000","last_undegraded":"2024-09-17T05:35:25.957097+0000","last_fullsized":"2024-09-17T05:35:25.957097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1234,"log_dups_size":0,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4196027,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1229,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968584+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:35:25.968584+0000","last_peered":"2024-09-17T05:35:25.968584+0000","last_clean":"2024-09-17T05:35:25.968584+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:35:25.968584+0000","last_undegraded":"2024-09-17T05:35:25.968584+0000","last_fullsized":"2024-09-17T05:35:25.968584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'6124","reported_seq":6173,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.242706+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:36:19.242706+0000","last_peered":"2024-09-17T05:36:19.242706+0000","last_clean":"2024-09-17T05:36:19.242706+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:36:19.242706+0000","last_undegraded":"2024-09-17T05:36:19.242706+0000","last_fullsized":"2024-09-17T05:36:19.242706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465673452,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":92183,"num_write_kb":1140422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.935850+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:35:25.935850+0000","last_peered":"2024-09-17T05:35:25.935850+0000","last_clean":"2024-09-17T05:35:25.935850+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:35:25.935850+0000","last_undegraded":"2024-09-17T05:35:25.935850+0000","last_fullsized":"2024-09-17T05:35:25.935850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.960667+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:35:25.960667+0000","last_peered":"2024-09-17T05:35:25.960667+0000","last_clean":"2024-09-17T05:35:25.960667+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:35:25.960667+0000","last_undegraded":"2024-09-17T05:35:25.960667+0000","last_fullsized":"2024-09-17T05:35:25.960667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"28'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335365+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:35:26.335365+0000","last_peered":"2024-09-17T05:35:26.335365+0000","last_clean":"2024-09-17T05:35:26.335365+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:35:26.335365+0000","last_undegraded":"2024-09-17T05:35:26.335365+0000","last_fullsized":"2024-09-17T05:35:26.335365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.876152+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:35:55.876152+0000","last_peered":"2024-09-17T05:35:55.876152+0000","last_clean":"2024-09-17T05:35:55.876152+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:35:55.876152+0000","last_undegraded":"2024-09-17T05:35:55.876152+0000","last_fullsized":"2024-09-17T05:35:55.876152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559514+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:36:06.559514+0000","last_peered":"2024-09-17T05:36:06.559514+0000","last_clean":"2024-09-17T05:36:06.559514+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:36:06.559514+0000","last_undegraded":"2024-09-17T05:36:06.559514+0000","last_fullsized":"2024-09-17T05:36:06.559514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559546+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:36:06.559546+0000","last_peered":"2024-09-17T05:36:06.559546+0000","last_clean":"2024-09-17T05:36:06.559546+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:36:06.559546+0000","last_undegraded":"2024-09-17T05:36:06.559546+0000","last_fullsized":"2024-09-17T05:36:06.559546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936653+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:35:25.936653+0000","last_peered":"2024-09-17T05:35:25.936653+0000","last_clean":"2024-09-17T05:35:25.936653+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:35:25.936653+0000","last_undegraded":"2024-09-17T05:35:25.936653+0000","last_fullsized":"2024-09-17T05:35:25.936653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"28'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952118+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:35:25.952118+0000","last_peered":"2024-09-17T05:35:25.952118+0000","last_clean":"2024-09-17T05:35:25.952118+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:35:25.952118+0000","last_undegraded":"2024-09-17T05:35:25.952118+0000","last_fullsized":"2024-09-17T05:35:25.952118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.193551+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:35:26.193551+0000","last_peered":"2024-09-17T05:35:26.193551+0000","last_clean":"2024-09-17T05:35:26.193551+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:35:26.193551+0000","last_undegraded":"2024-09-17T05:35:26.193551+0000","last_fullsized":"2024-09-17T05:35:26.193551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"31'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.061407+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:36:22.061407+0000","last_peered":"2024-09-17T05:36:22.061407+0000","last_clean":"2024-09-17T05:36:22.061407+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:36:22.061407+0000","last_undegraded":"2024-09-17T05:36:22.061407+0000","last_fullsized":"2024-09-17T05:36:22.061407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968478+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:35:25.968478+0000","last_peered":"2024-09-17T05:35:25.968478+0000","last_clean":"2024-09-17T05:35:25.968478+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:35:25.968478+0000","last_undegraded":"2024-09-17T05:35:25.968478+0000","last_fullsized":"2024-09-17T05:35:25.968478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941600+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:35:25.941600+0000","last_peered":"2024-09-17T05:35:25.941600+0000","last_clean":"2024-09-17T05:35:25.941600+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:35:25.941600+0000","last_undegraded":"2024-09-17T05:35:25.941600+0000","last_fullsized":"2024-09-17T05:35:25.941600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"31'7","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943098+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:35:25.943098+0000","last_peered":"2024-09-17T05:35:25.943098+0000","last_clean":"2024-09-17T05:35:25.943098+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:35:25.943098+0000","last_undegraded":"2024-09-17T05:35:25.943098+0000","last_fullsized":"2024-09-17T05:35:25.943098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065935+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:35:26.065935+0000","last_peered":"2024-09-17T05:35:26.065935+0000","last_clean":"2024-09-17T05:35:26.065935+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:35:26.065935+0000","last_undegraded":"2024-09-17T05:35:26.065935+0000","last_fullsized":"2024-09-17T05:35:26.065935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"28'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.948268+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:35:25.948268+0000","last_peered":"2024-09-17T05:35:25.948268+0000","last_clean":"2024-09-17T05:35:25.948268+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:35:25.948268+0000","last_undegraded":"2024-09-17T05:35:25.948268+0000","last_fullsized":"2024-09-17T05:35:25.948268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.168925+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:35:26.168925+0000","last_peered":"2024-09-17T05:35:26.168925+0000","last_clean":"2024-09-17T05:35:26.168925+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:35:26.168925+0000","last_undegraded":"2024-09-17T05:35:26.168925+0000","last_fullsized":"2024-09-17T05:35:26.168925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.193404+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:35:26.193404+0000","last_peered":"2024-09-17T05:35:26.193404+0000","last_clean":"2024-09-17T05:35:26.193404+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:35:26.193404+0000","last_undegraded":"2024-09-17T05:35:26.193404+0000","last_fullsized":"2024-09-17T05:35:26.193404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"28'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.951216+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:35:25.951216+0000","last_peered":"2024-09-17T05:35:25.951216+0000","last_clean":"2024-09-17T05:35:25.951216+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:35:25.951216+0000","last_undegraded":"2024-09-17T05:35:25.951216+0000","last_fullsized":"2024-09-17T05:35:25.951216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970393+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:35:25.970393+0000","last_peered":"2024-09-17T05:35:25.970393+0000","last_clean":"2024-09-17T05:35:25.970393+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:35:25.970393+0000","last_undegraded":"2024-09-17T05:35:25.970393+0000","last_fullsized":"2024-09-17T05:35:25.970393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997235+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:35:25.997235+0000","last_peered":"2024-09-17T05:35:25.997235+0000","last_clean":"2024-09-17T05:35:25.997235+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:35:25.997235+0000","last_undegraded":"2024-09-17T05:35:25.997235+0000","last_fullsized":"2024-09-17T05:35:25.997235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"28'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.942921+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:35:25.942921+0000","last_peered":"2024-09-17T05:35:25.942921+0000","last_clean":"2024-09-17T05:35:25.942921+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:35:25.942921+0000","last_undegraded":"2024-09-17T05:35:25.942921+0000","last_fullsized":"2024-09-17T05:35:25.942921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"28'11","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943835+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:35:25.943835+0000","last_peered":"2024-09-17T05:35:25.943835+0000","last_clean":"2024-09-17T05:35:25.943835+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:35:25.943835+0000","last_undegraded":"2024-09-17T05:35:25.943835+0000","last_fullsized":"2024-09-17T05:35:25.943835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943813+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:35:25.943813+0000","last_peered":"2024-09-17T05:35:25.943813+0000","last_clean":"2024-09-17T05:35:25.943813+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:35:25.943813+0000","last_undegraded":"2024-09-17T05:35:25.943813+0000","last_fullsized":"2024-09-17T05:35:25.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"31'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.976708+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:35:25.976708+0000","last_peered":"2024-09-17T05:35:25.976708+0000","last_clean":"2024-09-17T05:35:25.976708+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:35:25.976708+0000","last_undegraded":"2024-09-17T05:35:25.976708+0000","last_fullsized":"2024-09-17T05:35:25.976708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"28'3","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949226+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:35:25.949226+0000","last_peered":"2024-09-17T05:35:25.949226+0000","last_clean":"2024-09-17T05:35:25.949226+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:35:25.949226+0000","last_undegraded":"2024-09-17T05:35:25.949226+0000","last_fullsized":"2024-09-17T05:35:25.949226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"31'4","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943027+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:35:25.943027+0000","last_peered":"2024-09-17T05:35:25.943027+0000","last_clean":"2024-09-17T05:35:25.943027+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:35:25.943027+0000","last_undegraded":"2024-09-17T05:35:25.943027+0000","last_fullsized":"2024-09-17T05:35:25.943027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947757+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:35:25.947757+0000","last_peered":"2024-09-17T05:35:25.947757+0000","last_clean":"2024-09-17T05:35:25.947757+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:35:25.947757+0000","last_undegraded":"2024-09-17T05:35:25.947757+0000","last_fullsized":"2024-09-17T05:35:25.947757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"28'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943661+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:35:25.943661+0000","last_peered":"2024-09-17T05:35:25.943661+0000","last_clean":"2024-09-17T05:35:25.943661+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:35:25.943661+0000","last_undegraded":"2024-09-17T05:35:25.943661+0000","last_fullsized":"2024-09-17T05:35:25.943661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"28'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.935304+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:35:25.935304+0000","last_peered":"2024-09-17T05:35:25.935304+0000","last_clean":"2024-09-17T05:35:25.935304+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:35:25.935304+0000","last_undegraded":"2024-09-17T05:35:25.935304+0000","last_fullsized":"2024-09-17T05:35:25.935304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:40.792244+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:35:40.792244+0000","last_peered":"2024-09-17T05:35:40.792244+0000","last_clean":"2024-09-17T05:35:40.792244+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:35:40.792244+0000","last_undegraded":"2024-09-17T05:35:40.792244+0000","last_fullsized":"2024-09-17T05:35:40.792244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"28'2","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.969435+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:35:25.969435+0000","last_peered":"2024-09-17T05:35:25.969435+0000","last_clean":"2024-09-17T05:35:25.969435+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:35:25.969435+0000","last_undegraded":"2024-09-17T05:35:25.969435+0000","last_fullsized":"2024-09-17T05:35:25.969435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"28'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947901+0000","last_change":"2024-09-17T05:28:07.483393+0000","last_active":"2024-09-17T05:35:25.947901+0000","last_peered":"2024-09-17T05:35:25.947901+0000","last_clean":"2024-09-17T05:35:25.947901+0000","last_became_active":"2024-09-17T05:28:07.483003+0000","last_became_peered":"2024-09-17T05:28:07.483003+0000","last_unstale":"2024-09-17T05:35:25.947901+0000","last_undegraded":"2024-09-17T05:35:25.947901+0000","last_fullsized":"2024-09-17T05:35:25.947901+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":11805228873,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":525,"num_read_kb":0,"num_write":2618859,"num_write_kb":32721427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12280737792,"data_stored":23555850240,"data_compressed":1201049330,"data_compressed_allocated":4969824256,"data_compressed_original":16244936704,"omap_allocated":0,"internal_metadata":0},"log_size":174780,"ondisk_log_size":174780,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":82155219,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":346,"num_read_kb":78280,"num_write":2707,"num_write_kb":82299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33984512,"data_stored":164186953,"data_compressed":18806052,"data_compressed_allocated":21831680,"data_compressed_original":152186880,"omap_allocated":0,"internal_metadata":0},"log_size":2529,"ondisk_log_size":2529,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509550,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3211116,"kb_used_data":2047708,"kb_used_omap":1,"kb_used_meta":1163390,"kb_avail":91160724,"statfs":{"total":96636764160,"available":93348581376,"internally_reserved":0,"allocated":2096852992,"data_stored":4054768616,"data_compressed":208583112,"data_compressed_allocated":849747968,"data_compressed_original":2807865344,"omap_allocated":1593,"internal_metadata":1191311815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,0,0,0,0,0,0,30,93,11],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509552,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2199544,"kb_used_data":1403112,"kb_used_omap":1,"kb_used_meta":796414,"kb_avail":92172296,"statfs":{"total":96636764160,"available":94384431104,"internally_reserved":0,"allocated":1436786688,"data_stored":2758514067,"data_compressed":141474862,"data_compressed_allocated":581439488,"data_compressed_original":1903366144,"omap_allocated":1595,"internal_metadata":815528389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509552,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2990224,"kb_used_data":1907136,"kb_used_omap":1,"kb_used_meta":1083070,"kb_avail":91381616,"statfs":{"total":96636764160,"available":93574774784,"internally_reserved":0,"allocated":1952907264,"data_stored":3778817967,"data_compressed":194628104,"data_compressed_allocated":793247744,"data_compressed_original":2619359232,"omap_allocated":1595,"internal_metadata":1109064133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509553,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1602876,"kb_used_data":917612,"kb_used_omap":1,"kb_used_meta":685246,"kb_avail":92768964,"statfs":{"total":96636764160,"available":94995419136,"internally_reserved":0,"allocated":939634688,"data_stored":1833875455,"data_compressed":95381766,"data_compressed_allocated":384143360,"data_compressed_original":1278590976,"omap_allocated":1595,"internal_metadata":701692357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542254,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2975728,"kb_used_data":1898656,"kb_used_omap":1,"kb_used_meta":1077054,"kb_avail":91396112,"statfs":{"total":96636764160,"available":93589618688,"internally_reserved":0,"allocated":1944223744,"data_stored":3742764326,"data_compressed":190498180,"data_compressed_allocated":783704064,"data_compressed_original":2582458368,"omap_allocated":1590,"internal_metadata":1102903754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542255,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2517912,"kb_used_data":1445448,"kb_used_omap":1,"kb_used_meta":1072446,"kb_avail":91853928,"statfs":{"total":96636764160,"available":94058422272,"internally_reserved":0,"allocated":1480138752,"data_stored":2846927826,"data_compressed":145986734,"data_compressed_allocated":601456640,"data_compressed_original":1968443392,"omap_allocated":1594,"internal_metadata":1098185158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2478492,"kb_used_data":1631372,"kb_used_omap":1,"kb_used_meta":847102,"kb_avail":91893348,"statfs":{"total":96636764160,"available":94098788352,"internally_reserved":0,"allocated":1670524928,"data_stored":3211504529,"data_compressed":163583899,"data_compressed_allocated":673783808,"data_compressed_original":2214989824,"omap_allocated":1594,"internal_metadata":867432902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2167944,"kb_used_data":1320952,"kb_used_omap":1,"kb_used_meta":846974,"kb_avail":92203896,"statfs":{"total":96636764160,"available":94416789504,"internally_reserved":0,"allocated":1352654848,"data_stored":2583291230,"data_compressed":133191837,"data_compressed_allocated":544722944,"data_compressed_original":1775570944,"omap_allocated":1595,"internal_metadata":867301829},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5005312,"data_stored":27563310,"data_compressed":3259739,"data_compressed_allocated":3784704,"data_compressed_original":26361856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":23225030,"data_compressed":2717358,"data_compressed_allocated":3166208,"data_compressed_original":22245376,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":12583618,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":37751063,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":12585152,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6688768,"data_stored":25247853,"data_compressed":2684105,"data_compressed_allocated":3153920,"data_compressed_original":21733376,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":12585217,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2154496,"data_stored":12645710,"data_compressed":1495438,"data_compressed_allocated":1753088,"data_compressed_original":12271616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":844763136,"data_stored":1632104448,"data_compressed":83506478,"data_compressed_allocated":344780800,"data_compressed_original":1132122112,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1789231104,"data_stored":3444797440,"data_compressed":175919946,"data_compressed_allocated":726069248,"data_compressed_original":2381635584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1422041088,"data_stored":2724814848,"data_compressed":138920823,"data_compressed_allocated":574853120,"data_compressed_original":1877626880,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2066087936,"data_stored":3956056064,"data_compressed":201398317,"data_compressed_allocated":834166784,"data_compressed_original":2724134912,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1544409088,"data_stored":2963894272,"data_compressed":151060241,"data_compressed_allocated":624812032,"data_compressed_original":2044297216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1331101696,"data_stored":2527948800,"data_compressed":128951423,"data_compressed_allocated":535752704,"data_compressed_original":1732599808,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1368395776,"data_stored":2639163392,"data_compressed":135161769,"data_compressed_allocated":558776320,"data_compressed_original":1829543936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1914707968,"data_stored":3667070976,"data_compressed":186130333,"data_compressed_allocated":770613248,"data_compressed_original":2522976256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:36:29.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:36:29.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.932+0000 7f63e0121640 1 -- 172.21.15.117:0/125132840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c00037a0 msgr2=0x7f63c0003bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.932+0000 7f63e0121640 1 --2- 172.21.15.117:0/125132840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c00037a0 0x7f63c0003bf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f63d8072dc0 tx=0x7f63c0005830 comp rx=0 tx=0).stop 2024-09-17T05:36:29.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.933+0000 7f63e0121640 1 -- 172.21.15.117:0/125132840 shutdown_connections 2024-09-17T05:36:29.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.933+0000 7f63e0121640 1 --2- 172.21.15.117:0/125132840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c00037a0 0x7f63c0003bf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.933+0000 7f63e0121640 1 -- 172.21.15.117:0/125132840 >> 172.21.15.117:0/125132840 conn(0x7f63d806d190 msgr2=0x7f63d806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:29.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.933+0000 7f63e0121640 1 -- 172.21.15.117:0/125132840 shutdown_connections 2024-09-17T05:36:29.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.933+0000 7f63e0121640 1 -- 172.21.15.117:0/125132840 wait complete. 2024-09-17T05:36:29.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.937+0000 7f63e0121640 1 Processor -- start 2024-09-17T05:36:29.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.941+0000 7f63e0121640 1 -- start start 2024-09-17T05:36:29.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.943+0000 7f63e0121640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63c00037a0 0x7f63d813bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.943+0000 7f63e0121640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f63d8140f60 0x7f63d8203280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.943+0000 7f63e0121640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 0x7f63d8166360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.943+0000 7f63e0121640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f63d80c67f0 con 0x7f63d813e330 2024-09-17T05:36:29.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.943+0000 7f63e0121640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f63d80c3820 con 0x7f63c00037a0 2024-09-17T05:36:29.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.943+0000 7f63e0121640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f63d8151610 con 0x7f63d8140f60 2024-09-17T05:36:29.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.944+0000 7f63de6be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 0x7f63d8166360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.944+0000 7f63de6be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 0x7f63d8166360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60478/0 (socket says 172.21.15.117:60478) 2024-09-17T05:36:29.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.944+0000 7f63de6be640 1 -- 172.21.15.117:0/134239970 learned_addr learned my addr 172.21.15.117:0/134239970 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:29.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.944+0000 7f63dd6bc640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f63d8140f60 0x7f63d8203280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.944+0000 7f63ddebd640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63c00037a0 0x7f63d813bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.946+0000 7f63de6be640 1 -- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f63d8140f60 msgr2=0x7f63d8203280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.946+0000 7f63de6be640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f63d8140f60 0x7f63d8203280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.946+0000 7f63de6be640 1 -- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63c00037a0 msgr2=0x7f63d813bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:29.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.946+0000 7f63de6be640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63c00037a0 0x7f63d813bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:29.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.946+0000 7f63de6be640 1 -- 172.21.15.117:0/134239970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d82037c0 con 0x7f63d813e330 2024-09-17T05:36:29.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.946+0000 7f63de6be640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 0x7f63d8166360 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f63d80c4660 tx=0x7f63d000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:29.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.949+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d00035f0 con 0x7f63d813e330 2024-09-17T05:36:29.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.949+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d8141ef0 con 0x7f63d813e330 2024-09-17T05:36:29.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.949+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d8202a80 con 0x7f63d813e330 2024-09-17T05:36:29.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.959+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63d0003880 con 0x7f63d813e330 2024-09-17T05:36:29.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.964+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0003c90 con 0x7f63d813e330 2024-09-17T05:36:29.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.965+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f63d0011070 con 0x7f63d813e330 2024-09-17T05:36:29.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.966+0000 7f63d67fc640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f63c4078bc0 0x7f63c407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:29.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.966+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f63d009a5b0 con 0x7f63d813e330 2024-09-17T05:36:29.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.967+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b8005290 con 0x7f63d813e330 2024-09-17T05:36:29.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.967+0000 7f63ddebd640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f63c4078bc0 0x7f63c407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:29.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.980+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f63d0019060 con 0x7f63d813e330 2024-09-17T05:36:29.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:29.985+0000 7f63ddebd640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f63c4078bc0 0x7f63c407b0d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f63d806c490 tx=0x7f63cc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:30.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.335+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63b8000bc0 con 0x7f63c4078bc0 2024-09-17T05:36:30.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.347+0000 7f63d67fc640 1 -- 172.21.15.117:0/134239970 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+335532 (secure 0 0 0) 0x7f63b8000bc0 con 0x7f63c4078bc0 2024-09-17T05:36:30.349 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:36:30.359 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:36:30.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f63c4078bc0 msgr2=0x7f63c407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:30.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f63c4078bc0 0x7f63c407b0d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f63d806c490 tx=0x7f63cc008040 comp rx=0 tx=0).stop 2024-09-17T05:36:30.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 msgr2=0x7f63d8166360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:30.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 0x7f63d8166360 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f63d80c4660 tx=0x7f63d000c820 comp rx=0 tx=0).stop 2024-09-17T05:36:30.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 shutdown_connections 2024-09-17T05:36:30.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f63c4078bc0 0x7f63c407b0d0 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f63d806c490 tx=0x7f63cc008040 comp rx=0 tx=0).stop 2024-09-17T05:36:30.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63d813e330 0x7f63d8166360 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:30.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f63d8140f60 0x7f63d8203280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:30.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 --2- 172.21.15.117:0/134239970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63c00037a0 0x7f63d813bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:30.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.357+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 >> 172.21.15.117:0/134239970 conn(0x7f63d806d190 msgr2=0x7f63d80c78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:30.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.359+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 shutdown_connections 2024-09-17T05:36:30.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.363+0000 7f63e0121640 1 -- 172.21.15.117:0/134239970 wait complete. 2024-09-17T05:36:30.390 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-17T05:36:29.528242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11887974460,"num_objects":3096,"num_object_clones":0,"num_object_copies":12202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":937,"num_read_kb":78335,"num_write":2621667,"num_write_kb":32805354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177852,"ondisk_log_size":177852,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":20143836,"kb_used_data":12571996,"kb_used_omap":12,"kb_used_meta":7571699,"kb_avail":734830884,"statfs":{"total":773094113280,"available":752466825216,"internally_reserved":0,"allocated":12873723904,"data_stored":24810464016,"data_compressed":1273328494,"data_compressed_allocated":5212246016,"data_compressed_original":17150644224,"omap_allocated":12751,"internal_metadata":7753420337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,2,2,6,5,5,0,0,30,93,11],"upper_bound":16384},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1617300524,"num_objects":-386,"num_object_clones":0,"num_object_copies":-1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-386,"num_whiteouts":0,"num_read":93,"num_read_kb":12,"num_write":38695,"num_write_kb":174965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003492"},"pg_stats":[{"pgid":"2.1f","version":"30'18","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:45.149722+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:35:45.149722+0000","last_peered":"2024-09-17T05:35:45.149722+0000","last_clean":"2024-09-17T05:35:45.149722+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:35:45.149722+0000","last_undegraded":"2024-09-17T05:35:45.149722+0000","last_fullsized":"2024-09-17T05:35:45.149722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":4124,"num_write":15,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970029+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:35:25.970029+0000","last_peered":"2024-09-17T05:35:25.970029+0000","last_clean":"2024-09-17T05:35:25.970029+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:35:25.970029+0000","last_undegraded":"2024-09-17T05:35:25.970029+0000","last_fullsized":"2024-09-17T05:35:25.970029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"31'5749","reported_seq":5796,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:23.369232+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:36:23.369232+0000","last_peered":"2024-09-17T05:36:23.369232+0000","last_clean":"2024-09-17T05:36:23.369232+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:36:23.369232+0000","last_undegraded":"2024-09-17T05:36:23.369232+0000","last_fullsized":"2024-09-17T05:36:23.369232+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440520717,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":86546,"num_write_kb":1098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"29'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.956723+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:35:30.956723+0000","last_peered":"2024-09-17T05:35:30.956723+0000","last_clean":"2024-09-17T05:35:30.956723+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:35:30.956723+0000","last_undegraded":"2024-09-17T05:35:30.956723+0000","last_fullsized":"2024-09-17T05:35:30.956723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952279+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:35:25.952279+0000","last_peered":"2024-09-17T05:35:25.952279+0000","last_clean":"2024-09-17T05:35:25.952279+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:35:25.952279+0000","last_undegraded":"2024-09-17T05:35:25.952279+0000","last_fullsized":"2024-09-17T05:35:25.952279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"31'5672","reported_seq":5720,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.170725+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:36:25.170725+0000","last_peered":"2024-09-17T05:36:25.170725+0000","last_clean":"2024-09-17T05:36:25.170725+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:36:25.170725+0000","last_undegraded":"2024-09-17T05:36:25.170725+0000","last_fullsized":"2024-09-17T05:36:25.170725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402678953,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":85001,"num_write_kb":1045061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"31'49","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.841015+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:35:50.841015+0000","last_peered":"2024-09-17T05:35:50.841015+0000","last_clean":"2024-09-17T05:35:50.841015+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:35:50.841015+0000","last_undegraded":"2024-09-17T05:35:50.841015+0000","last_fullsized":"2024-09-17T05:35:50.841015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":8212,"num_write":40,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.969942+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:35:25.969942+0000","last_peered":"2024-09-17T05:35:25.969942+0000","last_clean":"2024-09-17T05:35:25.969942+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:35:25.969942+0000","last_undegraded":"2024-09-17T05:35:25.969942+0000","last_fullsized":"2024-09-17T05:35:25.969942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'5296","reported_seq":5345,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.978235+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:36:24.978235+0000","last_peered":"2024-09-17T05:36:24.978235+0000","last_clean":"2024-09-17T05:36:24.978235+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:36:24.978235+0000","last_undegraded":"2024-09-17T05:36:24.978235+0000","last_fullsized":"2024-09-17T05:36:24.978235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398475913,"num_objects":105,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79368,"num_write_kb":982831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"31'198","reported_seq":239,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.752100+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:36:26.752100+0000","last_peered":"2024-09-17T05:36:26.752100+0000","last_clean":"2024-09-17T05:36:26.752100+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:36:26.752100+0000","last_undegraded":"2024-09-17T05:36:26.752100+0000","last_fullsized":"2024-09-17T05:36:26.752100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395069,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":83,"num_write":197,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065126+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:35:26.065126+0000","last_peered":"2024-09-17T05:35:26.065126+0000","last_clean":"2024-09-17T05:35:26.065126+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:35:26.065126+0000","last_undegraded":"2024-09-17T05:35:26.065126+0000","last_fullsized":"2024-09-17T05:35:26.065126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"31'4547","reported_seq":4594,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.247886+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:36:25.247886+0000","last_peered":"2024-09-17T05:36:25.247886+0000","last_clean":"2024-09-17T05:36:25.247886+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:36:25.247886+0000","last_undegraded":"2024-09-17T05:36:25.247886+0000","last_fullsized":"2024-09-17T05:36:25.247886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348151202,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":68171,"num_write_kb":860103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335717+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:35:26.335717+0000","last_peered":"2024-09-17T05:35:26.335717+0000","last_clean":"2024-09-17T05:35:26.335717+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:35:26.335717+0000","last_undegraded":"2024-09-17T05:35:26.335717+0000","last_fullsized":"2024-09-17T05:35:26.335717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"31'126","reported_seq":169,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.838914+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:36:19.838914+0000","last_peered":"2024-09-17T05:36:19.838914+0000","last_clean":"2024-09-17T05:36:19.838914+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:36:19.838914+0000","last_undegraded":"2024-09-17T05:36:19.838914+0000","last_fullsized":"2024-09-17T05:36:19.838914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4104,"num_write":119,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"31'5574","reported_seq":5615,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.846827+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:36:22.846827+0000","last_peered":"2024-09-17T05:36:22.846827+0000","last_clean":"2024-09-17T05:36:22.846827+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:36:22.846827+0000","last_undegraded":"2024-09-17T05:36:22.846827+0000","last_fullsized":"2024-09-17T05:36:22.846827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419575348,"num_objects":107,"num_object_clones":0,"num_object_copies":428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":83636,"num_write_kb":1023293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947658+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:35:25.947658+0000","last_peered":"2024-09-17T05:35:25.947658+0000","last_clean":"2024-09-17T05:35:25.947658+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:35:25.947658+0000","last_undegraded":"2024-09-17T05:35:25.947658+0000","last_fullsized":"2024-09-17T05:35:25.947658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"31'6","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.663343+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:35:34.663343+0000","last_peered":"2024-09-17T05:35:34.663343+0000","last_clean":"2024-09-17T05:35:34.663343+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:35:34.663343+0000","last_undegraded":"2024-09-17T05:35:34.663343+0000","last_fullsized":"2024-09-17T05:35:34.663343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"31'5484","reported_seq":5529,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.331686+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:36:28.331686+0000","last_peered":"2024-09-17T05:36:28.331686+0000","last_clean":"2024-09-17T05:36:28.331686+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:36:28.331686+0000","last_undegraded":"2024-09-17T05:36:28.331686+0000","last_fullsized":"2024-09-17T05:36:28.331686+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268458584,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":81905,"num_write_kb":1031804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.012651+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:35:26.012651+0000","last_peered":"2024-09-17T05:35:26.012651+0000","last_clean":"2024-09-17T05:35:26.012651+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:35:26.012651+0000","last_undegraded":"2024-09-17T05:35:26.012651+0000","last_fullsized":"2024-09-17T05:35:26.012651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"30'4","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953261+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:35:25.953261+0000","last_peered":"2024-09-17T05:35:25.953261+0000","last_clean":"2024-09-17T05:35:25.953261+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:35:25.953261+0000","last_undegraded":"2024-09-17T05:35:25.953261+0000","last_fullsized":"2024-09-17T05:35:25.953261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'6469","reported_seq":6509,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.260774+0000","last_change":"2024-09-17T05:28:07.483827+0000","last_active":"2024-09-17T05:36:28.260774+0000","last_peered":"2024-09-17T05:36:28.260774+0000","last_clean":"2024-09-17T05:36:28.260774+0000","last_became_active":"2024-09-17T05:28:07.483263+0000","last_became_peered":"2024-09-17T05:28:07.483263+0000","last_unstale":"2024-09-17T05:36:28.260774+0000","last_undegraded":"2024-09-17T05:36:28.260774+0000","last_fullsized":"2024-09-17T05:36:28.260774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.655059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306281587,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":96582,"num_write_kb":1196264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2,3],"acting":[0,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.074006+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:35:26.074006+0000","last_peered":"2024-09-17T05:35:26.074006+0000","last_clean":"2024-09-17T05:35:26.074006+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:35:26.074006+0000","last_undegraded":"2024-09-17T05:35:26.074006+0000","last_fullsized":"2024-09-17T05:35:26.074006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"30'13","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.893052+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:35:31.893052+0000","last_peered":"2024-09-17T05:35:31.893052+0000","last_clean":"2024-09-17T05:35:31.893052+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:35:31.893052+0000","last_undegraded":"2024-09-17T05:35:31.893052+0000","last_fullsized":"2024-09-17T05:35:31.893052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":12,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"31'4761","reported_seq":4805,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.831627+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:36:24.831627+0000","last_peered":"2024-09-17T05:36:24.831627+0000","last_clean":"2024-09-17T05:36:24.831627+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:36:24.831627+0000","last_undegraded":"2024-09-17T05:36:24.831627+0000","last_fullsized":"2024-09-17T05:36:24.831627+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360925861,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":71329,"num_write_kb":889518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"28'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.940299+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:35:25.940299+0000","last_peered":"2024-09-17T05:35:25.940299+0000","last_clean":"2024-09-17T05:35:25.940299+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:35:25.940299+0000","last_undegraded":"2024-09-17T05:35:25.940299+0000","last_fullsized":"2024-09-17T05:35:25.940299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:04.068629+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:36:04.068629+0000","last_peered":"2024-09-17T05:36:04.068629+0000","last_clean":"2024-09-17T05:36:04.068629+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:36:04.068629+0000","last_undegraded":"2024-09-17T05:36:04.068629+0000","last_fullsized":"2024-09-17T05:36:04.068629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"31'5448","reported_seq":5497,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.113608+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:36:25.113608+0000","last_peered":"2024-09-17T05:36:25.113608+0000","last_clean":"2024-09-17T05:36:25.113608+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:36:25.113608+0000","last_undegraded":"2024-09-17T05:36:25.113608+0000","last_fullsized":"2024-09-17T05:36:25.113608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402664508,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":81667,"num_write_kb":1039133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"28'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.961318+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:35:25.961318+0000","last_peered":"2024-09-17T05:35:25.961318+0000","last_clean":"2024-09-17T05:35:25.961318+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:35:25.961318+0000","last_undegraded":"2024-09-17T05:35:25.961318+0000","last_fullsized":"2024-09-17T05:35:25.961318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"31'214","reported_seq":258,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.682203+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:36:24.682203+0000","last_peered":"2024-09-17T05:36:24.682203+0000","last_clean":"2024-09-17T05:36:24.682203+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:36:24.682203+0000","last_undegraded":"2024-09-17T05:36:24.682203+0000","last_fullsized":"2024-09-17T05:36:24.682203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":21,"num_write":434,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"31'5919","reported_seq":5970,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.298032+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:36:25.298032+0000","last_peered":"2024-09-17T05:36:25.298032+0000","last_clean":"2024-09-17T05:36:25.298032+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:36:25.298032+0000","last_undegraded":"2024-09-17T05:36:25.298032+0000","last_fullsized":"2024-09-17T05:36:25.298032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5919,"log_dups_size":0,"ondisk_log_size":5919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411056495,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":88791,"num_write_kb":1123429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.573647+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:35:55.573647+0000","last_peered":"2024-09-17T05:35:55.573647+0000","last_clean":"2024-09-17T05:35:55.573647+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:35:55.573647+0000","last_undegraded":"2024-09-17T05:35:55.573647+0000","last_fullsized":"2024-09-17T05:35:55.573647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.066159+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:35:26.066159+0000","last_peered":"2024-09-17T05:35:26.066159+0000","last_clean":"2024-09-17T05:35:26.066159+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:35:26.066159+0000","last_undegraded":"2024-09-17T05:35:26.066159+0000","last_fullsized":"2024-09-17T05:35:26.066159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'5008","reported_seq":5050,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.326050+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:36:27.326050+0000","last_peered":"2024-09-17T05:36:27.326050+0000","last_clean":"2024-09-17T05:36:27.326050+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:36:27.326050+0000","last_undegraded":"2024-09-17T05:36:27.326050+0000","last_fullsized":"2024-09-17T05:36:27.326050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276833139,"num_objects":70,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":74950,"num_write_kb":939196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"31'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:59.340987+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:35:59.340987+0000","last_peered":"2024-09-17T05:35:59.340987+0000","last_clean":"2024-09-17T05:35:59.340987+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:35:59.340987+0000","last_undegraded":"2024-09-17T05:35:59.340987+0000","last_fullsized":"2024-09-17T05:35:59.340987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'1","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968299+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:35:25.968299+0000","last_peered":"2024-09-17T05:35:25.968299+0000","last_clean":"2024-09-17T05:35:25.968299+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:35:25.968299+0000","last_undegraded":"2024-09-17T05:35:25.968299+0000","last_fullsized":"2024-09-17T05:35:25.968299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"31'6130","reported_seq":6178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.692134+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:36:27.692134+0000","last_peered":"2024-09-17T05:36:27.692134+0000","last_clean":"2024-09-17T05:36:27.692134+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:36:27.692134+0000","last_undegraded":"2024-09-17T05:36:27.692134+0000","last_fullsized":"2024-09-17T05:36:27.692134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381700523,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":91961,"num_write_kb":1145505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.950313+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:36:27.950313+0000","last_peered":"2024-09-17T05:36:27.950313+0000","last_clean":"2024-09-17T05:36:27.950313+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:36:27.950313+0000","last_undegraded":"2024-09-17T05:36:27.950313+0000","last_fullsized":"2024-09-17T05:36:27.950313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"30'41","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.332894+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:35:32.332894+0000","last_peered":"2024-09-17T05:35:32.332894+0000","last_clean":"2024-09-17T05:35:32.332894+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:35:32.332894+0000","last_undegraded":"2024-09-17T05:35:32.332894+0000","last_fullsized":"2024-09-17T05:35:32.332894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":35,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'5280","reported_seq":5330,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.478014+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:36:27.478014+0000","last_peered":"2024-09-17T05:36:27.478014+0000","last_clean":"2024-09-17T05:36:27.478014+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:36:27.478014+0000","last_undegraded":"2024-09-17T05:36:27.478014+0000","last_fullsized":"2024-09-17T05:36:27.478014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310455443,"num_objects":80,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":78953,"num_write_kb":985615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949408+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:35:25.949408+0000","last_peered":"2024-09-17T05:35:25.949408+0000","last_clean":"2024-09-17T05:35:25.949408+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:35:25.949408+0000","last_undegraded":"2024-09-17T05:35:25.949408+0000","last_fullsized":"2024-09-17T05:35:25.949408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"31'27","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.695352+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:36:24.695352+0000","last_peered":"2024-09-17T05:36:24.695352+0000","last_clean":"2024-09-17T05:36:24.695352+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:36:24.695352+0000","last_undegraded":"2024-09-17T05:36:24.695352+0000","last_fullsized":"2024-09-17T05:36:24.695352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61980,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":83,"num_write":24,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'5837","reported_seq":5888,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.309862+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:36:27.309862+0000","last_peered":"2024-09-17T05:36:27.309862+0000","last_clean":"2024-09-17T05:36:27.309862+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:36:27.309862+0000","last_undegraded":"2024-09-17T05:36:27.309862+0000","last_fullsized":"2024-09-17T05:36:27.309862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356569031,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":87303,"num_write_kb":1093347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962251+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:35:25.962251+0000","last_peered":"2024-09-17T05:35:25.962251+0000","last_clean":"2024-09-17T05:35:25.962251+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:35:25.962251+0000","last_undegraded":"2024-09-17T05:35:25.962251+0000","last_fullsized":"2024-09-17T05:35:25.962251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"30'91","reported_seq":136,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.303333+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:35:31.303333+0000","last_peered":"2024-09-17T05:35:31.303333+0000","last_clean":"2024-09-17T05:35:31.303333+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:35:31.303333+0000","last_undegraded":"2024-09-17T05:35:31.303333+0000","last_fullsized":"2024-09-17T05:35:31.303333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8204,"num_write":88,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"31'4749","reported_seq":4791,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.976651+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:36:26.976651+0000","last_peered":"2024-09-17T05:36:26.976651+0000","last_clean":"2024-09-17T05:36:26.976651+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:36:26.976651+0000","last_undegraded":"2024-09-17T05:36:26.976651+0000","last_fullsized":"2024-09-17T05:36:26.976651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4749,"log_dups_size":0,"ondisk_log_size":4749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294034242,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":70943,"num_write_kb":889582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.940713+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:35:25.940713+0000","last_peered":"2024-09-17T05:35:25.940713+0000","last_clean":"2024-09-17T05:35:25.940713+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:35:25.940713+0000","last_undegraded":"2024-09-17T05:35:25.940713+0000","last_fullsized":"2024-09-17T05:35:25.940713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.201104+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:35:26.201104+0000","last_peered":"2024-09-17T05:35:26.201104+0000","last_clean":"2024-09-17T05:35:26.201104+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:35:26.201104+0000","last_undegraded":"2024-09-17T05:35:26.201104+0000","last_fullsized":"2024-09-17T05:35:26.201104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":135,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187694+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:35:26.187694+0000","last_peered":"2024-09-17T05:35:26.187694+0000","last_clean":"2024-09-17T05:35:26.187694+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:35:26.187694+0000","last_undegraded":"2024-09-17T05:35:26.187694+0000","last_fullsized":"2024-09-17T05:35:26.187694+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'5318","reported_seq":5367,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.734064+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:36:24.734064+0000","last_peered":"2024-09-17T05:36:24.734064+0000","last_clean":"2024-09-17T05:36:24.734064+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:36:24.734064+0000","last_undegraded":"2024-09-17T05:36:24.734064+0000","last_fullsized":"2024-09-17T05:36:24.734064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398468612,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":79949,"num_write_kb":1015697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065270+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:35:26.065270+0000","last_peered":"2024-09-17T05:35:26.065270+0000","last_clean":"2024-09-17T05:35:26.065270+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:35:26.065270+0000","last_undegraded":"2024-09-17T05:35:26.065270+0000","last_fullsized":"2024-09-17T05:35:26.065270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"30'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.981638+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:35:25.981638+0000","last_peered":"2024-09-17T05:35:25.981638+0000","last_clean":"2024-09-17T05:35:25.981638+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:35:25.981638+0000","last_undegraded":"2024-09-17T05:35:25.981638+0000","last_fullsized":"2024-09-17T05:35:25.981638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"31'5485","reported_seq":5533,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.259874+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:36:25.259874+0000","last_peered":"2024-09-17T05:36:25.259874+0000","last_clean":"2024-09-17T05:36:25.259874+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:36:25.259874+0000","last_undegraded":"2024-09-17T05:36:25.259874+0000","last_fullsized":"2024-09-17T05:36:25.259874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373295017,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82225,"num_write_kb":1025442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943229+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:35:25.943229+0000","last_peered":"2024-09-17T05:35:25.943229+0000","last_clean":"2024-09-17T05:35:25.943229+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:35:25.943229+0000","last_undegraded":"2024-09-17T05:35:25.943229+0000","last_fullsized":"2024-09-17T05:35:25.943229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"30'20","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.072706+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:35:32.072706+0000","last_peered":"2024-09-17T05:35:32.072706+0000","last_clean":"2024-09-17T05:35:32.072706+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:35:32.072706+0000","last_undegraded":"2024-09-17T05:35:32.072706+0000","last_fullsized":"2024-09-17T05:35:32.072706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":19,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"31'5002","reported_seq":5050,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.107012+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:36:27.107012+0000","last_peered":"2024-09-17T05:36:27.107012+0000","last_clean":"2024-09-17T05:36:27.107012+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:36:27.107012+0000","last_undegraded":"2024-09-17T05:36:27.107012+0000","last_fullsized":"2024-09-17T05:36:27.107012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314598708,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":74882,"num_write_kb":933970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.946846+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:35:25.946846+0000","last_peered":"2024-09-17T05:35:25.946846+0000","last_clean":"2024-09-17T05:35:25.946846+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:35:25.946846+0000","last_undegraded":"2024-09-17T05:35:25.946846+0000","last_fullsized":"2024-09-17T05:35:25.946846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'27","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.808293+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:35:50.808293+0000","last_peered":"2024-09-17T05:35:50.808293+0000","last_clean":"2024-09-17T05:35:50.808293+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:35:50.808293+0000","last_undegraded":"2024-09-17T05:35:50.808293+0000","last_fullsized":"2024-09-17T05:35:50.808293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":26,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"31'5083","reported_seq":5128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:20.292547+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:36:20.292547+0000","last_peered":"2024-09-17T05:36:20.292547+0000","last_clean":"2024-09-17T05:36:20.292547+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:36:20.292547+0000","last_undegraded":"2024-09-17T05:36:20.292547+0000","last_fullsized":"2024-09-17T05:36:20.292547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394284666,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76376,"num_write_kb":955028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936873+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:35:25.936873+0000","last_peered":"2024-09-17T05:35:25.936873+0000","last_clean":"2024-09-17T05:35:25.936873+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:35:25.936873+0000","last_undegraded":"2024-09-17T05:35:25.936873+0000","last_fullsized":"2024-09-17T05:35:25.936873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"31'8","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.592840+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:35:34.592840+0000","last_peered":"2024-09-17T05:35:34.592840+0000","last_clean":"2024-09-17T05:35:34.592840+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:35:34.592840+0000","last_undegraded":"2024-09-17T05:35:34.592840+0000","last_fullsized":"2024-09-17T05:35:34.592840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"31'5113","reported_seq":5158,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.336370+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:36:26.336370+0000","last_peered":"2024-09-17T05:36:26.336370+0000","last_clean":"2024-09-17T05:36:26.336370+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:36:26.336370+0000","last_undegraded":"2024-09-17T05:36:26.336370+0000","last_fullsized":"2024-09-17T05:36:26.336370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344012237,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76601,"num_write_kb":970760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.073783+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:35:26.073783+0000","last_peered":"2024-09-17T05:35:26.073783+0000","last_clean":"2024-09-17T05:35:26.073783+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:35:26.073783+0000","last_undegraded":"2024-09-17T05:35:26.073783+0000","last_fullsized":"2024-09-17T05:35:26.073783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"30'19","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.833437+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:35:50.833437+0000","last_peered":"2024-09-17T05:35:50.833437+0000","last_clean":"2024-09-17T05:35:50.833437+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:35:50.833437+0000","last_undegraded":"2024-09-17T05:35:50.833437+0000","last_fullsized":"2024-09-17T05:35:50.833437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":40,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"31'5738","reported_seq":5789,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.859102+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:36:26.859102+0000","last_peered":"2024-09-17T05:36:26.859102+0000","last_clean":"2024-09-17T05:36:26.859102+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:36:26.859102+0000","last_undegraded":"2024-09-17T05:36:26.859102+0000","last_fullsized":"2024-09-17T05:36:26.859102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331603745,"num_objects":90,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":85828,"num_write_kb":1086559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187433+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:35:26.187433+0000","last_peered":"2024-09-17T05:35:26.187433+0000","last_clean":"2024-09-17T05:35:26.187433+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:35:26.187433+0000","last_undegraded":"2024-09-17T05:35:26.187433+0000","last_fullsized":"2024-09-17T05:35:26.187433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"30'13","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943104+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:35:25.943104+0000","last_peered":"2024-09-17T05:35:25.943104+0000","last_clean":"2024-09-17T05:35:25.943104+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:35:25.943104+0000","last_undegraded":"2024-09-17T05:35:25.943104+0000","last_fullsized":"2024-09-17T05:35:25.943104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1689,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":13,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"31'5839","reported_seq":5887,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.046398+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:36:27.046398+0000","last_peered":"2024-09-17T05:36:27.046398+0000","last_clean":"2024-09-17T05:36:27.046398+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:36:27.046398+0000","last_undegraded":"2024-09-17T05:36:27.046398+0000","last_fullsized":"2024-09-17T05:36:27.046398+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339798246,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87347,"num_write_kb":1070021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.200625+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:35:26.200625+0000","last_peered":"2024-09-17T05:35:26.200625+0000","last_clean":"2024-09-17T05:35:26.200625+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:35:26.200625+0000","last_undegraded":"2024-09-17T05:35:26.200625+0000","last_fullsized":"2024-09-17T05:35:26.200625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"30'6","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335841+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:35:26.335841+0000","last_peered":"2024-09-17T05:35:26.335841+0000","last_clean":"2024-09-17T05:35:26.335841+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:35:26.335841+0000","last_undegraded":"2024-09-17T05:35:26.335841+0000","last_fullsized":"2024-09-17T05:35:26.335841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'5863","reported_seq":5910,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.484944+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:36:27.484944+0000","last_peered":"2024-09-17T05:36:27.484944+0000","last_clean":"2024-09-17T05:36:27.484944+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:36:27.484944+0000","last_undegraded":"2024-09-17T05:36:27.484944+0000","last_fullsized":"2024-09-17T05:36:27.484944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5863,"log_dups_size":0,"ondisk_log_size":5863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318999243,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87526,"num_write_kb":1085839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"28'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970702+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:35:25.970702+0000","last_peered":"2024-09-17T05:35:25.970702+0000","last_clean":"2024-09-17T05:35:25.970702+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:35:25.970702+0000","last_undegraded":"2024-09-17T05:35:25.970702+0000","last_fullsized":"2024-09-17T05:35:25.970702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"30'87","reported_seq":134,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.501205+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:35:31.501205+0000","last_peered":"2024-09-17T05:35:31.501205+0000","last_clean":"2024-09-17T05:35:31.501205+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:35:31.501205+0000","last_undegraded":"2024-09-17T05:35:31.501205+0000","last_fullsized":"2024-09-17T05:35:31.501205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":8218,"num_write":88,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"31'5132","reported_seq":5178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.833860+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:36:22.833860+0000","last_peered":"2024-09-17T05:36:22.833860+0000","last_clean":"2024-09-17T05:36:22.833860+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:36:22.833860+0000","last_undegraded":"2024-09-17T05:36:22.833860+0000","last_fullsized":"2024-09-17T05:36:22.833860+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394264576,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":76895,"num_write_kb":960359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.945356+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:35:25.945356+0000","last_peered":"2024-09-17T05:35:25.945356+0000","last_clean":"2024-09-17T05:35:25.945356+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:35:25.945356+0000","last_undegraded":"2024-09-17T05:35:25.945356+0000","last_fullsized":"2024-09-17T05:35:25.945356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"31'213","reported_seq":258,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:15.763813+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:36:15.763813+0000","last_peered":"2024-09-17T05:36:15.763813+0000","last_clean":"2024-09-17T05:36:15.763813+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:36:15.763813+0000","last_undegraded":"2024-09-17T05:36:15.763813+0000","last_fullsized":"2024-09-17T05:36:15.763813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":212,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"31'5323","reported_seq":5372,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.196028+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:36:25.196028+0000","last_peered":"2024-09-17T05:36:25.196028+0000","last_clean":"2024-09-17T05:36:25.196028+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:36:25.196028+0000","last_undegraded":"2024-09-17T05:36:25.196028+0000","last_fullsized":"2024-09-17T05:36:25.196028+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390153058,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79495,"num_write_kb":1008039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997367+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:35:25.997367+0000","last_peered":"2024-09-17T05:35:25.997367+0000","last_clean":"2024-09-17T05:35:25.997367+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:35:25.997367+0000","last_undegraded":"2024-09-17T05:35:25.997367+0000","last_fullsized":"2024-09-17T05:35:25.997367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"31'10","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.839506+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:35:50.839506+0000","last_peered":"2024-09-17T05:35:50.839506+0000","last_clean":"2024-09-17T05:35:50.839506+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:35:50.839506+0000","last_undegraded":"2024-09-17T05:35:50.839506+0000","last_fullsized":"2024-09-17T05:35:50.839506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"31'5584","reported_seq":5628,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.389841+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:36:25.389841+0000","last_peered":"2024-09-17T05:36:25.389841+0000","last_clean":"2024-09-17T05:36:25.389841+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:36:25.389841+0000","last_undegraded":"2024-09-17T05:36:25.389841+0000","last_fullsized":"2024-09-17T05:36:25.389841+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390177777,"num_objects":103,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":15,"num_read_kb":0,"num_write":83378,"num_write_kb":1022097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952372+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:35:25.952372+0000","last_peered":"2024-09-17T05:35:25.952372+0000","last_clean":"2024-09-17T05:35:25.952372+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:35:25.952372+0000","last_undegraded":"2024-09-17T05:35:25.952372+0000","last_fullsized":"2024-09-17T05:35:25.952372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"30'3","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943930+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:35:25.943930+0000","last_peered":"2024-09-17T05:35:25.943930+0000","last_clean":"2024-09-17T05:35:25.943930+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:35:25.943930+0000","last_undegraded":"2024-09-17T05:35:25.943930+0000","last_fullsized":"2024-09-17T05:35:25.943930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"31'5306","reported_seq":5349,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.902604+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:36:24.902604+0000","last_peered":"2024-09-17T05:36:24.902604+0000","last_clean":"2024-09-17T05:36:24.902604+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:36:24.902604+0000","last_undegraded":"2024-09-17T05:36:24.902604+0000","last_fullsized":"2024-09-17T05:36:24.902604+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385944448,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":13,"num_read_kb":0,"num_write":79626,"num_write_kb":1002963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936737+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:35:25.936737+0000","last_peered":"2024-09-17T05:35:25.936737+0000","last_clean":"2024-09-17T05:35:25.936737+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:35:25.936737+0000","last_undegraded":"2024-09-17T05:35:25.936737+0000","last_fullsized":"2024-09-17T05:35:25.936737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"31'11","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.972204+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:35:25.972204+0000","last_peered":"2024-09-17T05:35:25.972204+0000","last_clean":"2024-09-17T05:35:25.972204+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:35:25.972204+0000","last_undegraded":"2024-09-17T05:35:25.972204+0000","last_fullsized":"2024-09-17T05:35:25.972204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'5506","reported_seq":5553,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.983518+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:36:24.983518+0000","last_peered":"2024-09-17T05:36:24.983518+0000","last_clean":"2024-09-17T05:36:24.983518+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:36:24.983518+0000","last_undegraded":"2024-09-17T05:36:24.983518+0000","last_fullsized":"2024-09-17T05:36:24.983518+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406877841,"num_objects":103,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82610,"num_write_kb":1020969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6379","reported_seq":6427,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.313591+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:36:25.313591+0000","last_peered":"2024-09-17T05:36:25.313591+0000","last_clean":"2024-09-17T05:36:25.313591+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:36:25.313591+0000","last_undegraded":"2024-09-17T05:36:25.313591+0000","last_fullsized":"2024-09-17T05:36:25.313591+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452996234,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":95842,"num_write_kb":1202164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953061+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:35:25.953061+0000","last_peered":"2024-09-17T05:35:25.953061+0000","last_clean":"2024-09-17T05:35:25.953061+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:35:25.953061+0000","last_undegraded":"2024-09-17T05:35:25.953061+0000","last_fullsized":"2024-09-17T05:35:25.953061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"30'27","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.703337+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:35:31.703337+0000","last_peered":"2024-09-17T05:35:31.703337+0000","last_clean":"2024-09-17T05:35:31.703337+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:35:31.703337+0000","last_undegraded":"2024-09-17T05:35:31.703337+0000","last_fullsized":"2024-09-17T05:35:31.703337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":24,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"31'5525","reported_seq":5577,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:23.122693+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:36:23.122693+0000","last_peered":"2024-09-17T05:36:23.122693+0000","last_clean":"2024-09-17T05:36:23.122693+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:36:23.122693+0000","last_undegraded":"2024-09-17T05:36:23.122693+0000","last_fullsized":"2024-09-17T05:36:23.122693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427902156,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":82949,"num_write_kb":1044536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"28'14","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949598+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:35:25.949598+0000","last_peered":"2024-09-17T05:35:25.949598+0000","last_clean":"2024-09-17T05:35:25.949598+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:35:25.949598+0000","last_undegraded":"2024-09-17T05:35:25.949598+0000","last_fullsized":"2024-09-17T05:35:25.949598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"30'5","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962325+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:35:25.962325+0000","last_peered":"2024-09-17T05:35:25.962325+0000","last_clean":"2024-09-17T05:35:25.962325+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:35:25.962325+0000","last_undegraded":"2024-09-17T05:35:25.962325+0000","last_fullsized":"2024-09-17T05:35:25.962325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"31'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.941350+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:35:30.941350+0000","last_peered":"2024-09-17T05:35:30.941350+0000","last_clean":"2024-09-17T05:35:30.941350+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:35:30.941350+0000","last_undegraded":"2024-09-17T05:35:30.941350+0000","last_fullsized":"2024-09-17T05:35:30.941350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4108,"num_write":17,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941766+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:35:25.941766+0000","last_peered":"2024-09-17T05:35:25.941766+0000","last_clean":"2024-09-17T05:35:25.941766+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:35:25.941766+0000","last_undegraded":"2024-09-17T05:35:25.941766+0000","last_fullsized":"2024-09-17T05:35:25.941766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4537","reported_seq":4575,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.435603+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:36:25.435603+0000","last_peered":"2024-09-17T05:36:25.435603+0000","last_clean":"2024-09-17T05:36:25.435603+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:36:25.435603+0000","last_undegraded":"2024-09-17T05:36:25.435603+0000","last_fullsized":"2024-09-17T05:36:25.435603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297797311,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":68041,"num_write_kb":833128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1234","reported_seq":1275,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.957097+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:35:25.957097+0000","last_peered":"2024-09-17T05:35:25.957097+0000","last_clean":"2024-09-17T05:35:25.957097+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:35:25.957097+0000","last_undegraded":"2024-09-17T05:35:25.957097+0000","last_fullsized":"2024-09-17T05:35:25.957097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1234,"log_dups_size":0,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4196027,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1229,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968584+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:35:25.968584+0000","last_peered":"2024-09-17T05:35:25.968584+0000","last_clean":"2024-09-17T05:35:25.968584+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:35:25.968584+0000","last_undegraded":"2024-09-17T05:35:25.968584+0000","last_fullsized":"2024-09-17T05:35:25.968584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'6124","reported_seq":6173,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.242706+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:36:19.242706+0000","last_peered":"2024-09-17T05:36:19.242706+0000","last_clean":"2024-09-17T05:36:19.242706+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:36:19.242706+0000","last_undegraded":"2024-09-17T05:36:19.242706+0000","last_fullsized":"2024-09-17T05:36:19.242706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465673452,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":92183,"num_write_kb":1140422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.935850+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:35:25.935850+0000","last_peered":"2024-09-17T05:35:25.935850+0000","last_clean":"2024-09-17T05:35:25.935850+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:35:25.935850+0000","last_undegraded":"2024-09-17T05:35:25.935850+0000","last_fullsized":"2024-09-17T05:35:25.935850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.960667+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:35:25.960667+0000","last_peered":"2024-09-17T05:35:25.960667+0000","last_clean":"2024-09-17T05:35:25.960667+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:35:25.960667+0000","last_undegraded":"2024-09-17T05:35:25.960667+0000","last_fullsized":"2024-09-17T05:35:25.960667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"28'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335365+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:35:26.335365+0000","last_peered":"2024-09-17T05:35:26.335365+0000","last_clean":"2024-09-17T05:35:26.335365+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:35:26.335365+0000","last_undegraded":"2024-09-17T05:35:26.335365+0000","last_fullsized":"2024-09-17T05:35:26.335365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.876152+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:35:55.876152+0000","last_peered":"2024-09-17T05:35:55.876152+0000","last_clean":"2024-09-17T05:35:55.876152+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:35:55.876152+0000","last_undegraded":"2024-09-17T05:35:55.876152+0000","last_fullsized":"2024-09-17T05:35:55.876152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559514+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:36:06.559514+0000","last_peered":"2024-09-17T05:36:06.559514+0000","last_clean":"2024-09-17T05:36:06.559514+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:36:06.559514+0000","last_undegraded":"2024-09-17T05:36:06.559514+0000","last_fullsized":"2024-09-17T05:36:06.559514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559546+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:36:06.559546+0000","last_peered":"2024-09-17T05:36:06.559546+0000","last_clean":"2024-09-17T05:36:06.559546+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:36:06.559546+0000","last_undegraded":"2024-09-17T05:36:06.559546+0000","last_fullsized":"2024-09-17T05:36:06.559546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936653+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:35:25.936653+0000","last_peered":"2024-09-17T05:35:25.936653+0000","last_clean":"2024-09-17T05:35:25.936653+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:35:25.936653+0000","last_undegraded":"2024-09-17T05:35:25.936653+0000","last_fullsized":"2024-09-17T05:35:25.936653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"28'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952118+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:35:25.952118+0000","last_peered":"2024-09-17T05:35:25.952118+0000","last_clean":"2024-09-17T05:35:25.952118+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:35:25.952118+0000","last_undegraded":"2024-09-17T05:35:25.952118+0000","last_fullsized":"2024-09-17T05:35:25.952118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.193551+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:35:26.193551+0000","last_peered":"2024-09-17T05:35:26.193551+0000","last_clean":"2024-09-17T05:35:26.193551+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:35:26.193551+0000","last_undegraded":"2024-09-17T05:35:26.193551+0000","last_fullsized":"2024-09-17T05:35:26.193551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"31'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.061407+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:36:22.061407+0000","last_peered":"2024-09-17T05:36:22.061407+0000","last_clean":"2024-09-17T05:36:22.061407+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:36:22.061407+0000","last_undegraded":"2024-09-17T05:36:22.061407+0000","last_fullsized":"2024-09-17T05:36:22.061407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968478+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:35:25.968478+0000","last_peered":"2024-09-17T05:35:25.968478+0000","last_clean":"2024-09-17T05:35:25.968478+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:35:25.968478+0000","last_undegraded":"2024-09-17T05:35:25.968478+0000","last_fullsized":"2024-09-17T05:35:25.968478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941600+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:35:25.941600+0000","last_peered":"2024-09-17T05:35:25.941600+0000","last_clean":"2024-09-17T05:35:25.941600+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:35:25.941600+0000","last_undegraded":"2024-09-17T05:35:25.941600+0000","last_fullsized":"2024-09-17T05:35:25.941600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"31'7","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943098+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:35:25.943098+0000","last_peered":"2024-09-17T05:35:25.943098+0000","last_clean":"2024-09-17T05:35:25.943098+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:35:25.943098+0000","last_undegraded":"2024-09-17T05:35:25.943098+0000","last_fullsized":"2024-09-17T05:35:25.943098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065935+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:35:26.065935+0000","last_peered":"2024-09-17T05:35:26.065935+0000","last_clean":"2024-09-17T05:35:26.065935+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:35:26.065935+0000","last_undegraded":"2024-09-17T05:35:26.065935+0000","last_fullsized":"2024-09-17T05:35:26.065935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"28'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.948268+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:35:25.948268+0000","last_peered":"2024-09-17T05:35:25.948268+0000","last_clean":"2024-09-17T05:35:25.948268+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:35:25.948268+0000","last_undegraded":"2024-09-17T05:35:25.948268+0000","last_fullsized":"2024-09-17T05:35:25.948268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.168925+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:35:26.168925+0000","last_peered":"2024-09-17T05:35:26.168925+0000","last_clean":"2024-09-17T05:35:26.168925+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:35:26.168925+0000","last_undegraded":"2024-09-17T05:35:26.168925+0000","last_fullsized":"2024-09-17T05:35:26.168925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.193404+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:35:26.193404+0000","last_peered":"2024-09-17T05:35:26.193404+0000","last_clean":"2024-09-17T05:35:26.193404+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:35:26.193404+0000","last_undegraded":"2024-09-17T05:35:26.193404+0000","last_fullsized":"2024-09-17T05:35:26.193404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"28'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.951216+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:35:25.951216+0000","last_peered":"2024-09-17T05:35:25.951216+0000","last_clean":"2024-09-17T05:35:25.951216+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:35:25.951216+0000","last_undegraded":"2024-09-17T05:35:25.951216+0000","last_fullsized":"2024-09-17T05:35:25.951216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970393+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:35:25.970393+0000","last_peered":"2024-09-17T05:35:25.970393+0000","last_clean":"2024-09-17T05:35:25.970393+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:35:25.970393+0000","last_undegraded":"2024-09-17T05:35:25.970393+0000","last_fullsized":"2024-09-17T05:35:25.970393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997235+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:35:25.997235+0000","last_peered":"2024-09-17T05:35:25.997235+0000","last_clean":"2024-09-17T05:35:25.997235+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:35:25.997235+0000","last_undegraded":"2024-09-17T05:35:25.997235+0000","last_fullsized":"2024-09-17T05:35:25.997235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"28'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.942921+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:35:25.942921+0000","last_peered":"2024-09-17T05:35:25.942921+0000","last_clean":"2024-09-17T05:35:25.942921+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:35:25.942921+0000","last_undegraded":"2024-09-17T05:35:25.942921+0000","last_fullsized":"2024-09-17T05:35:25.942921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"28'11","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943835+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:35:25.943835+0000","last_peered":"2024-09-17T05:35:25.943835+0000","last_clean":"2024-09-17T05:35:25.943835+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:35:25.943835+0000","last_undegraded":"2024-09-17T05:35:25.943835+0000","last_fullsized":"2024-09-17T05:35:25.943835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943813+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:35:25.943813+0000","last_peered":"2024-09-17T05:35:25.943813+0000","last_clean":"2024-09-17T05:35:25.943813+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:35:25.943813+0000","last_undegraded":"2024-09-17T05:35:25.943813+0000","last_fullsized":"2024-09-17T05:35:25.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"31'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.976708+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:35:25.976708+0000","last_peered":"2024-09-17T05:35:25.976708+0000","last_clean":"2024-09-17T05:35:25.976708+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:35:25.976708+0000","last_undegraded":"2024-09-17T05:35:25.976708+0000","last_fullsized":"2024-09-17T05:35:25.976708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"28'3","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949226+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:35:25.949226+0000","last_peered":"2024-09-17T05:35:25.949226+0000","last_clean":"2024-09-17T05:35:25.949226+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:35:25.949226+0000","last_undegraded":"2024-09-17T05:35:25.949226+0000","last_fullsized":"2024-09-17T05:35:25.949226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"31'4","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943027+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:35:25.943027+0000","last_peered":"2024-09-17T05:35:25.943027+0000","last_clean":"2024-09-17T05:35:25.943027+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:35:25.943027+0000","last_undegraded":"2024-09-17T05:35:25.943027+0000","last_fullsized":"2024-09-17T05:35:25.943027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947757+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:35:25.947757+0000","last_peered":"2024-09-17T05:35:25.947757+0000","last_clean":"2024-09-17T05:35:25.947757+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:35:25.947757+0000","last_undegraded":"2024-09-17T05:35:25.947757+0000","last_fullsized":"2024-09-17T05:35:25.947757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"28'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943661+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:35:25.943661+0000","last_peered":"2024-09-17T05:35:25.943661+0000","last_clean":"2024-09-17T05:35:25.943661+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:35:25.943661+0000","last_undegraded":"2024-09-17T05:35:25.943661+0000","last_fullsized":"2024-09-17T05:35:25.943661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"28'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.935304+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:35:25.935304+0000","last_peered":"2024-09-17T05:35:25.935304+0000","last_clean":"2024-09-17T05:35:25.935304+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:35:25.935304+0000","last_undegraded":"2024-09-17T05:35:25.935304+0000","last_fullsized":"2024-09-17T05:35:25.935304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:40.792244+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:35:40.792244+0000","last_peered":"2024-09-17T05:35:40.792244+0000","last_clean":"2024-09-17T05:35:40.792244+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:35:40.792244+0000","last_undegraded":"2024-09-17T05:35:40.792244+0000","last_fullsized":"2024-09-17T05:35:40.792244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"28'2","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.969435+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:35:25.969435+0000","last_peered":"2024-09-17T05:35:25.969435+0000","last_clean":"2024-09-17T05:35:25.969435+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:35:25.969435+0000","last_undegraded":"2024-09-17T05:35:25.969435+0000","last_fullsized":"2024-09-17T05:35:25.969435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"28'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947901+0000","last_change":"2024-09-17T05:28:07.483393+0000","last_active":"2024-09-17T05:35:25.947901+0000","last_peered":"2024-09-17T05:35:25.947901+0000","last_clean":"2024-09-17T05:35:25.947901+0000","last_became_active":"2024-09-17T05:28:07.483003+0000","last_became_peered":"2024-09-17T05:28:07.483003+0000","last_unstale":"2024-09-17T05:35:25.947901+0000","last_undegraded":"2024-09-17T05:35:25.947901+0000","last_fullsized":"2024-09-17T05:35:25.947901+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":11805228873,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":525,"num_read_kb":0,"num_write":2618859,"num_write_kb":32721427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12280737792,"data_stored":23555850240,"data_compressed":1201049330,"data_compressed_allocated":4969824256,"data_compressed_original":16244936704,"omap_allocated":0,"internal_metadata":0},"log_size":174780,"ondisk_log_size":174780,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":82155219,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":346,"num_read_kb":78280,"num_write":2707,"num_write_kb":82299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33984512,"data_stored":164186953,"data_compressed":18806052,"data_compressed_allocated":21831680,"data_compressed_original":152186880,"omap_allocated":0,"internal_metadata":0},"log_size":2529,"ondisk_log_size":2529,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509550,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3211116,"kb_used_data":2047708,"kb_used_omap":1,"kb_used_meta":1163390,"kb_avail":91160724,"statfs":{"total":96636764160,"available":93348581376,"internally_reserved":0,"allocated":2096852992,"data_stored":4054768616,"data_compressed":208583112,"data_compressed_allocated":849747968,"data_compressed_original":2807865344,"omap_allocated":1593,"internal_metadata":1191311815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,0,0,0,0,0,0,30,93,11],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509552,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2199544,"kb_used_data":1403112,"kb_used_omap":1,"kb_used_meta":796414,"kb_avail":92172296,"statfs":{"total":96636764160,"available":94384431104,"internally_reserved":0,"allocated":1436786688,"data_stored":2758514067,"data_compressed":141474862,"data_compressed_allocated":581439488,"data_compressed_original":1903366144,"omap_allocated":1595,"internal_metadata":815528389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509552,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2990224,"kb_used_data":1907136,"kb_used_omap":1,"kb_used_meta":1083070,"kb_avail":91381616,"statfs":{"total":96636764160,"available":93574774784,"internally_reserved":0,"allocated":1952907264,"data_stored":3778817967,"data_compressed":194628104,"data_compressed_allocated":793247744,"data_compressed_original":2619359232,"omap_allocated":1595,"internal_metadata":1109064133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509553,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1602876,"kb_used_data":917612,"kb_used_omap":1,"kb_used_meta":685246,"kb_avail":92768964,"statfs":{"total":96636764160,"available":94995419136,"internally_reserved":0,"allocated":939634688,"data_stored":1833875455,"data_compressed":95381766,"data_compressed_allocated":384143360,"data_compressed_original":1278590976,"omap_allocated":1595,"internal_metadata":701692357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542254,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2975728,"kb_used_data":1898656,"kb_used_omap":1,"kb_used_meta":1077054,"kb_avail":91396112,"statfs":{"total":96636764160,"available":93589618688,"internally_reserved":0,"allocated":1944223744,"data_stored":3742764326,"data_compressed":190498180,"data_compressed_allocated":783704064,"data_compressed_original":2582458368,"omap_allocated":1590,"internal_metadata":1102903754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542255,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2517912,"kb_used_data":1445448,"kb_used_omap":1,"kb_used_meta":1072446,"kb_avail":91853928,"statfs":{"total":96636764160,"available":94058422272,"internally_reserved":0,"allocated":1480138752,"data_stored":2846927826,"data_compressed":145986734,"data_compressed_allocated":601456640,"data_compressed_original":1968443392,"omap_allocated":1594,"internal_metadata":1098185158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2478492,"kb_used_data":1631372,"kb_used_omap":1,"kb_used_meta":847102,"kb_avail":91893348,"statfs":{"total":96636764160,"available":94098788352,"internally_reserved":0,"allocated":1670524928,"data_stored":3211504529,"data_compressed":163583899,"data_compressed_allocated":673783808,"data_compressed_original":2214989824,"omap_allocated":1594,"internal_metadata":867432902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2167944,"kb_used_data":1320952,"kb_used_omap":1,"kb_used_meta":846974,"kb_avail":92203896,"statfs":{"total":96636764160,"available":94416789504,"internally_reserved":0,"allocated":1352654848,"data_stored":2583291230,"data_compressed":133191837,"data_compressed_allocated":544722944,"data_compressed_original":1775570944,"omap_allocated":1595,"internal_metadata":867301829},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5005312,"data_stored":27563310,"data_compressed":3259739,"data_compressed_allocated":3784704,"data_compressed_original":26361856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":23225030,"data_compressed":2717358,"data_compressed_allocated":3166208,"data_compressed_original":22245376,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":12583618,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":37751063,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":12585152,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6688768,"data_stored":25247853,"data_compressed":2684105,"data_compressed_allocated":3153920,"data_compressed_original":21733376,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":12585217,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2154496,"data_stored":12645710,"data_compressed":1495438,"data_compressed_allocated":1753088,"data_compressed_original":12271616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":844763136,"data_stored":1632104448,"data_compressed":83506478,"data_compressed_allocated":344780800,"data_compressed_original":1132122112,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1789231104,"data_stored":3444797440,"data_compressed":175919946,"data_compressed_allocated":726069248,"data_compressed_original":2381635584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1422041088,"data_stored":2724814848,"data_compressed":138920823,"data_compressed_allocated":574853120,"data_compressed_original":1877626880,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2066087936,"data_stored":3956056064,"data_compressed":201398317,"data_compressed_allocated":834166784,"data_compressed_original":2724134912,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1544409088,"data_stored":2963894272,"data_compressed":151060241,"data_compressed_allocated":624812032,"data_compressed_original":2044297216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1331101696,"data_stored":2527948800,"data_compressed":128951423,"data_compressed_allocated":535752704,"data_compressed_original":1732599808,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1368395776,"data_stored":2639163392,"data_compressed":135161769,"data_compressed_allocated":558776320,"data_compressed_original":1829543936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1914707968,"data_stored":3667070976,"data_compressed":186130333,"data_compressed_allocated":770613248,"data_compressed_original":2522976256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:36:30.396 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T05:36:30.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:36:30.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.643+0000 7f88648d9640 1 -- 172.21.15.117:0/3993818656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 msgr2=0x7f8848003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:30.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.643+0000 7f88648d9640 1 --2- 172.21.15.117:0/3993818656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8848003ea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8854006fd0 tx=0x7f8848005830 comp rx=0 tx=0).stop 2024-09-17T05:36:30.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.654+0000 7f88648d9640 1 -- 172.21.15.117:0/3993818656 shutdown_connections 2024-09-17T05:36:30.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.654+0000 7f88648d9640 1 --2- 172.21.15.117:0/3993818656 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8848003ea0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:30.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.654+0000 7f88648d9640 1 -- 172.21.15.117:0/3993818656 >> 172.21.15.117:0/3993818656 conn(0x7f885804fe30 msgr2=0x7f8858050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:30.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.656+0000 7f88648d9640 1 -- 172.21.15.117:0/3993818656 shutdown_connections 2024-09-17T05:36:30.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.656+0000 7f88648d9640 1 -- 172.21.15.117:0/3993818656 wait complete. 2024-09-17T05:36:30.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.659+0000 7f88648d9640 1 Processor -- start 2024-09-17T05:36:30.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.659+0000 7f88648d9640 1 -- start start 2024-09-17T05:36:30.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.661+0000 7f88648d9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8858179720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.661+0000 7f88648d9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 0x7f88580cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8858179720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8858179720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:58660/0 (socket says 172.21.15.117:58660) 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f885e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 0x7f88580cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f885e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 0x7f88580cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60490/0 (socket says 172.21.15.117:60490) 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f885ed9d640 1 -- 172.21.15.117:0/2718133229 learned_addr learned my addr 172.21.15.117:0/2718133229 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f88648d9640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8858049ed0 0x7f88580cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f88648d9640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88580468b0 con 0x7f88580d5ef0 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.662+0000 7f88648d9640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8858045b90 con 0x7f8858049ed0 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.663+0000 7f88648d9640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f88580458c0 con 0x7f8848003a50 2024-09-17T05:36:30.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.663+0000 7f885f59e640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8858049ed0 0x7f88580cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885e59c640 1 -- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 msgr2=0x7f8858179720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885e59c640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8858179720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885e59c640 1 -- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8858049ed0 msgr2=0x7f88580cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885e59c640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8858049ed0 0x7f88580cf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885e59c640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8858047e60 con 0x7f88580d5ef0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885ed9d640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8858179720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.664+0000 7f885e59c640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 0x7f88580cdbd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f88580e3250 tx=0x7f8850009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.665+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850015020 con 0x7f88580d5ef0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.665+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8850011850 con 0x7f88580d5ef0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.665+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850011ca0 con 0x7f88580d5ef0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.665+0000 7f88648d9640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88580d5c10 con 0x7f88580d5ef0 2024-09-17T05:36:30.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.665+0000 7f88648d9640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8858047660 con 0x7f88580d5ef0 2024-09-17T05:36:30.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.666+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8838005290 con 0x7f88580d5ef0 2024-09-17T05:36:30.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.672+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8850018020 con 0x7f88580d5ef0 2024-09-17T05:36:30.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.673+0000 7f88477fe640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f884c078c60 0x7f884c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:30.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.673+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f885009af20 con 0x7f88580d5ef0 2024-09-17T05:36:30.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.682+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f885001f020 con 0x7f88580d5ef0 2024-09-17T05:36:30.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.697+0000 7f885ed9d640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f884c078c60 0x7f884c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:30.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:30.709+0000 7f885ed9d640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f884c078c60 0x7f884c07b170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8854000fd0 tx=0x7f8854038040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:31.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.043+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8838000bc0 con 0x7f884c078c60 2024-09-17T05:36:31.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.062+0000 7f88477fe640 1 -- 172.21.15.117:0/2718133229 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+335532 (secure 0 0 0) 0x7f8838000bc0 con 0x7f884c078c60 2024-09-17T05:36:31.066 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:36:31.068 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f884c078c60 msgr2=0x7f884c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f884c078c60 0x7f884c07b170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8854000fd0 tx=0x7f8854038040 comp rx=0 tx=0).stop 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 msgr2=0x7f88580cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 0x7f88580cdbd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f88580e3250 tx=0x7f8850009f90 comp rx=0 tx=0).stop 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 shutdown_connections 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f884c078c60 0x7f884c07b170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8858049ed0 0x7f88580cf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88580d5ef0 0x7f88580cdbd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 --2- 172.21.15.117:0/2718133229 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8848003a50 0x7f8858179720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 >> 172.21.15.117:0/2718133229 conn(0x7f885804fe30 msgr2=0x7f88580dc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 shutdown_connections 2024-09-17T05:36:31.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.068+0000 7f88457fa640 1 -- 172.21.15.117:0/2718133229 wait complete. 2024-09-17T05:36:31.087 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-17T05:36:29.528242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11887974460,"num_objects":3096,"num_object_clones":0,"num_object_copies":12202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":937,"num_read_kb":78335,"num_write":2621667,"num_write_kb":32805354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177852,"ondisk_log_size":177852,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":20143836,"kb_used_data":12571996,"kb_used_omap":12,"kb_used_meta":7571699,"kb_avail":734830884,"statfs":{"total":773094113280,"available":752466825216,"internally_reserved":0,"allocated":12873723904,"data_stored":24810464016,"data_compressed":1273328494,"data_compressed_allocated":5212246016,"data_compressed_original":17150644224,"omap_allocated":12751,"internal_metadata":7753420337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,2,2,6,5,5,0,0,30,93,11],"upper_bound":16384},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1617300524,"num_objects":-386,"num_object_clones":0,"num_object_copies":-1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-386,"num_whiteouts":0,"num_read":93,"num_read_kb":12,"num_write":38695,"num_write_kb":174965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003492"},"pg_stats":[{"pgid":"2.1f","version":"30'18","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:45.149722+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:35:45.149722+0000","last_peered":"2024-09-17T05:35:45.149722+0000","last_clean":"2024-09-17T05:35:45.149722+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:35:45.149722+0000","last_undegraded":"2024-09-17T05:35:45.149722+0000","last_fullsized":"2024-09-17T05:35:45.149722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":4124,"num_write":15,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970029+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:35:25.970029+0000","last_peered":"2024-09-17T05:35:25.970029+0000","last_clean":"2024-09-17T05:35:25.970029+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:35:25.970029+0000","last_undegraded":"2024-09-17T05:35:25.970029+0000","last_fullsized":"2024-09-17T05:35:25.970029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"31'5749","reported_seq":5796,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:23.369232+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:36:23.369232+0000","last_peered":"2024-09-17T05:36:23.369232+0000","last_clean":"2024-09-17T05:36:23.369232+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:36:23.369232+0000","last_undegraded":"2024-09-17T05:36:23.369232+0000","last_fullsized":"2024-09-17T05:36:23.369232+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440520717,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":86546,"num_write_kb":1098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"29'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.956723+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:35:30.956723+0000","last_peered":"2024-09-17T05:35:30.956723+0000","last_clean":"2024-09-17T05:35:30.956723+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:35:30.956723+0000","last_undegraded":"2024-09-17T05:35:30.956723+0000","last_fullsized":"2024-09-17T05:35:30.956723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952279+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:35:25.952279+0000","last_peered":"2024-09-17T05:35:25.952279+0000","last_clean":"2024-09-17T05:35:25.952279+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:35:25.952279+0000","last_undegraded":"2024-09-17T05:35:25.952279+0000","last_fullsized":"2024-09-17T05:35:25.952279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"31'5672","reported_seq":5720,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.170725+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:36:25.170725+0000","last_peered":"2024-09-17T05:36:25.170725+0000","last_clean":"2024-09-17T05:36:25.170725+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:36:25.170725+0000","last_undegraded":"2024-09-17T05:36:25.170725+0000","last_fullsized":"2024-09-17T05:36:25.170725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402678953,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":85001,"num_write_kb":1045061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"31'49","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.841015+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:35:50.841015+0000","last_peered":"2024-09-17T05:35:50.841015+0000","last_clean":"2024-09-17T05:35:50.841015+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:35:50.841015+0000","last_undegraded":"2024-09-17T05:35:50.841015+0000","last_fullsized":"2024-09-17T05:35:50.841015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:10.179479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":8212,"num_write":40,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.969942+0000","last_change":"2024-09-17T05:27:26.663983+0000","last_active":"2024-09-17T05:35:25.969942+0000","last_peered":"2024-09-17T05:35:25.969942+0000","last_clean":"2024-09-17T05:35:25.969942+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:35:25.969942+0000","last_undegraded":"2024-09-17T05:35:25.969942+0000","last_fullsized":"2024-09-17T05:35:25.969942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'5296","reported_seq":5345,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.978235+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:36:24.978235+0000","last_peered":"2024-09-17T05:36:24.978235+0000","last_clean":"2024-09-17T05:36:24.978235+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:36:24.978235+0000","last_undegraded":"2024-09-17T05:36:24.978235+0000","last_fullsized":"2024-09-17T05:36:24.978235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5296,"log_dups_size":0,"ondisk_log_size":5296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398475913,"num_objects":105,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79368,"num_write_kb":982831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"31'198","reported_seq":239,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.752100+0000","last_change":"2024-09-17T05:27:25.657145+0000","last_active":"2024-09-17T05:36:26.752100+0000","last_peered":"2024-09-17T05:36:26.752100+0000","last_clean":"2024-09-17T05:36:26.752100+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:36:26.752100+0000","last_undegraded":"2024-09-17T05:36:26.752100+0000","last_fullsized":"2024-09-17T05:36:26.752100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:48.969163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395069,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":83,"num_write":197,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065126+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:35:26.065126+0000","last_peered":"2024-09-17T05:35:26.065126+0000","last_clean":"2024-09-17T05:35:26.065126+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:35:26.065126+0000","last_undegraded":"2024-09-17T05:35:26.065126+0000","last_fullsized":"2024-09-17T05:35:26.065126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"31'4547","reported_seq":4594,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.247886+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:36:25.247886+0000","last_peered":"2024-09-17T05:36:25.247886+0000","last_clean":"2024-09-17T05:36:25.247886+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:36:25.247886+0000","last_undegraded":"2024-09-17T05:36:25.247886+0000","last_fullsized":"2024-09-17T05:36:25.247886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348151202,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":68171,"num_write_kb":860103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335717+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:35:26.335717+0000","last_peered":"2024-09-17T05:35:26.335717+0000","last_clean":"2024-09-17T05:35:26.335717+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:35:26.335717+0000","last_undegraded":"2024-09-17T05:35:26.335717+0000","last_fullsized":"2024-09-17T05:35:26.335717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"31'126","reported_seq":169,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.838914+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:36:19.838914+0000","last_peered":"2024-09-17T05:36:19.838914+0000","last_clean":"2024-09-17T05:36:19.838914+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:36:19.838914+0000","last_undegraded":"2024-09-17T05:36:19.838914+0000","last_fullsized":"2024-09-17T05:36:19.838914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4104,"num_write":119,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"31'5574","reported_seq":5615,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.846827+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:36:22.846827+0000","last_peered":"2024-09-17T05:36:22.846827+0000","last_clean":"2024-09-17T05:36:22.846827+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:36:22.846827+0000","last_undegraded":"2024-09-17T05:36:22.846827+0000","last_fullsized":"2024-09-17T05:36:22.846827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419575348,"num_objects":107,"num_object_clones":0,"num_object_copies":428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":83636,"num_write_kb":1023293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947658+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:35:25.947658+0000","last_peered":"2024-09-17T05:35:25.947658+0000","last_clean":"2024-09-17T05:35:25.947658+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:35:25.947658+0000","last_undegraded":"2024-09-17T05:35:25.947658+0000","last_fullsized":"2024-09-17T05:35:25.947658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"31'6","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.663343+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:35:34.663343+0000","last_peered":"2024-09-17T05:35:34.663343+0000","last_clean":"2024-09-17T05:35:34.663343+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:35:34.663343+0000","last_undegraded":"2024-09-17T05:35:34.663343+0000","last_fullsized":"2024-09-17T05:35:34.663343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"31'5484","reported_seq":5529,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.331686+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:36:28.331686+0000","last_peered":"2024-09-17T05:36:28.331686+0000","last_clean":"2024-09-17T05:36:28.331686+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:36:28.331686+0000","last_undegraded":"2024-09-17T05:36:28.331686+0000","last_fullsized":"2024-09-17T05:36:28.331686+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:40.785331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268458584,"num_objects":73,"num_object_clones":0,"num_object_copies":292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":81905,"num_write_kb":1031804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.012651+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:35:26.012651+0000","last_peered":"2024-09-17T05:35:26.012651+0000","last_clean":"2024-09-17T05:35:26.012651+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:35:26.012651+0000","last_undegraded":"2024-09-17T05:35:26.012651+0000","last_fullsized":"2024-09-17T05:35:26.012651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"30'4","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953261+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:35:25.953261+0000","last_peered":"2024-09-17T05:35:25.953261+0000","last_clean":"2024-09-17T05:35:25.953261+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:35:25.953261+0000","last_undegraded":"2024-09-17T05:35:25.953261+0000","last_fullsized":"2024-09-17T05:35:25.953261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'6469","reported_seq":6509,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.260774+0000","last_change":"2024-09-17T05:28:07.483827+0000","last_active":"2024-09-17T05:36:28.260774+0000","last_peered":"2024-09-17T05:36:28.260774+0000","last_clean":"2024-09-17T05:36:28.260774+0000","last_became_active":"2024-09-17T05:28:07.483263+0000","last_became_peered":"2024-09-17T05:28:07.483263+0000","last_unstale":"2024-09-17T05:36:28.260774+0000","last_undegraded":"2024-09-17T05:36:28.260774+0000","last_fullsized":"2024-09-17T05:36:28.260774+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.655059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306281587,"num_objects":78,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":96582,"num_write_kb":1196264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2,3],"acting":[0,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.074006+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:35:26.074006+0000","last_peered":"2024-09-17T05:35:26.074006+0000","last_clean":"2024-09-17T05:35:26.074006+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:35:26.074006+0000","last_undegraded":"2024-09-17T05:35:26.074006+0000","last_fullsized":"2024-09-17T05:35:26.074006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"30'13","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.893052+0000","last_change":"2024-09-17T05:27:25.660260+0000","last_active":"2024-09-17T05:35:31.893052+0000","last_peered":"2024-09-17T05:35:31.893052+0000","last_clean":"2024-09-17T05:35:31.893052+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:35:31.893052+0000","last_undegraded":"2024-09-17T05:35:31.893052+0000","last_fullsized":"2024-09-17T05:35:31.893052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:45:52.041361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":12,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"31'4761","reported_seq":4805,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.831627+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:36:24.831627+0000","last_peered":"2024-09-17T05:36:24.831627+0000","last_clean":"2024-09-17T05:36:24.831627+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:36:24.831627+0000","last_undegraded":"2024-09-17T05:36:24.831627+0000","last_fullsized":"2024-09-17T05:36:24.831627+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360925861,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":71329,"num_write_kb":889518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"28'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.940299+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:35:25.940299+0000","last_peered":"2024-09-17T05:35:25.940299+0000","last_clean":"2024-09-17T05:35:25.940299+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:35:25.940299+0000","last_undegraded":"2024-09-17T05:35:25.940299+0000","last_fullsized":"2024-09-17T05:35:25.940299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:04.068629+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:36:04.068629+0000","last_peered":"2024-09-17T05:36:04.068629+0000","last_clean":"2024-09-17T05:36:04.068629+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:36:04.068629+0000","last_undegraded":"2024-09-17T05:36:04.068629+0000","last_fullsized":"2024-09-17T05:36:04.068629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"31'5448","reported_seq":5497,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.113608+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:36:25.113608+0000","last_peered":"2024-09-17T05:36:25.113608+0000","last_clean":"2024-09-17T05:36:25.113608+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:36:25.113608+0000","last_undegraded":"2024-09-17T05:36:25.113608+0000","last_fullsized":"2024-09-17T05:36:25.113608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402664508,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":81667,"num_write_kb":1039133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"28'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.961318+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:35:25.961318+0000","last_peered":"2024-09-17T05:35:25.961318+0000","last_clean":"2024-09-17T05:35:25.961318+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:35:25.961318+0000","last_undegraded":"2024-09-17T05:35:25.961318+0000","last_fullsized":"2024-09-17T05:35:25.961318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"31'214","reported_seq":258,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.682203+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:36:24.682203+0000","last_peered":"2024-09-17T05:36:24.682203+0000","last_clean":"2024-09-17T05:36:24.682203+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:36:24.682203+0000","last_undegraded":"2024-09-17T05:36:24.682203+0000","last_fullsized":"2024-09-17T05:36:24.682203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":21,"num_write":434,"num_write_kb":176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"31'5919","reported_seq":5970,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.298032+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:36:25.298032+0000","last_peered":"2024-09-17T05:36:25.298032+0000","last_clean":"2024-09-17T05:36:25.298032+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:36:25.298032+0000","last_undegraded":"2024-09-17T05:36:25.298032+0000","last_fullsized":"2024-09-17T05:36:25.298032+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5919,"log_dups_size":0,"ondisk_log_size":5919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411056495,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":88791,"num_write_kb":1123429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.573647+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:35:55.573647+0000","last_peered":"2024-09-17T05:35:55.573647+0000","last_clean":"2024-09-17T05:35:55.573647+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:35:55.573647+0000","last_undegraded":"2024-09-17T05:35:55.573647+0000","last_fullsized":"2024-09-17T05:35:55.573647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.066159+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:35:26.066159+0000","last_peered":"2024-09-17T05:35:26.066159+0000","last_clean":"2024-09-17T05:35:26.066159+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:35:26.066159+0000","last_undegraded":"2024-09-17T05:35:26.066159+0000","last_fullsized":"2024-09-17T05:35:26.066159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'5008","reported_seq":5050,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.326050+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:36:27.326050+0000","last_peered":"2024-09-17T05:36:27.326050+0000","last_clean":"2024-09-17T05:36:27.326050+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:36:27.326050+0000","last_undegraded":"2024-09-17T05:36:27.326050+0000","last_fullsized":"2024-09-17T05:36:27.326050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276833139,"num_objects":70,"num_object_clones":0,"num_object_copies":280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":74950,"num_write_kb":939196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"31'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:59.340987+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:35:59.340987+0000","last_peered":"2024-09-17T05:35:59.340987+0000","last_clean":"2024-09-17T05:35:59.340987+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:35:59.340987+0000","last_undegraded":"2024-09-17T05:35:59.340987+0000","last_fullsized":"2024-09-17T05:35:59.340987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'1","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968299+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:35:25.968299+0000","last_peered":"2024-09-17T05:35:25.968299+0000","last_clean":"2024-09-17T05:35:25.968299+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:35:25.968299+0000","last_undegraded":"2024-09-17T05:35:25.968299+0000","last_fullsized":"2024-09-17T05:35:25.968299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"31'6130","reported_seq":6178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.692134+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:36:27.692134+0000","last_peered":"2024-09-17T05:36:27.692134+0000","last_clean":"2024-09-17T05:36:27.692134+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:36:27.692134+0000","last_undegraded":"2024-09-17T05:36:27.692134+0000","last_fullsized":"2024-09-17T05:36:27.692134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381700523,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":91961,"num_write_kb":1145505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.950313+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:36:27.950313+0000","last_peered":"2024-09-17T05:36:27.950313+0000","last_clean":"2024-09-17T05:36:27.950313+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:36:27.950313+0000","last_undegraded":"2024-09-17T05:36:27.950313+0000","last_fullsized":"2024-09-17T05:36:27.950313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:31.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"30'41","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.332894+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:35:32.332894+0000","last_peered":"2024-09-17T05:35:32.332894+0000","last_clean":"2024-09-17T05:35:32.332894+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:35:32.332894+0000","last_undegraded":"2024-09-17T05:35:32.332894+0000","last_fullsized":"2024-09-17T05:35:32.332894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":35,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'5280","reported_seq":5330,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.478014+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:36:27.478014+0000","last_peered":"2024-09-17T05:36:27.478014+0000","last_clean":"2024-09-17T05:36:27.478014+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:36:27.478014+0000","last_undegraded":"2024-09-17T05:36:27.478014+0000","last_fullsized":"2024-09-17T05:36:27.478014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310455443,"num_objects":80,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":78953,"num_write_kb":985615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949408+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:35:25.949408+0000","last_peered":"2024-09-17T05:35:25.949408+0000","last_clean":"2024-09-17T05:35:25.949408+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:35:25.949408+0000","last_undegraded":"2024-09-17T05:35:25.949408+0000","last_fullsized":"2024-09-17T05:35:25.949408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"31'27","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.695352+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:36:24.695352+0000","last_peered":"2024-09-17T05:36:24.695352+0000","last_clean":"2024-09-17T05:36:24.695352+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:36:24.695352+0000","last_undegraded":"2024-09-17T05:36:24.695352+0000","last_fullsized":"2024-09-17T05:36:24.695352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61980,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":83,"num_write":24,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'5837","reported_seq":5888,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.309862+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:36:27.309862+0000","last_peered":"2024-09-17T05:36:27.309862+0000","last_clean":"2024-09-17T05:36:27.309862+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:36:27.309862+0000","last_undegraded":"2024-09-17T05:36:27.309862+0000","last_fullsized":"2024-09-17T05:36:27.309862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356569031,"num_objects":92,"num_object_clones":0,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":87303,"num_write_kb":1093347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962251+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:35:25.962251+0000","last_peered":"2024-09-17T05:35:25.962251+0000","last_clean":"2024-09-17T05:35:25.962251+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:35:25.962251+0000","last_undegraded":"2024-09-17T05:35:25.962251+0000","last_fullsized":"2024-09-17T05:35:25.962251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"30'91","reported_seq":136,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.303333+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:35:31.303333+0000","last_peered":"2024-09-17T05:35:31.303333+0000","last_clean":"2024-09-17T05:35:31.303333+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:35:31.303333+0000","last_undegraded":"2024-09-17T05:35:31.303333+0000","last_fullsized":"2024-09-17T05:35:31.303333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8204,"num_write":88,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"31'4749","reported_seq":4791,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.976651+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:36:26.976651+0000","last_peered":"2024-09-17T05:36:26.976651+0000","last_clean":"2024-09-17T05:36:26.976651+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:36:26.976651+0000","last_undegraded":"2024-09-17T05:36:26.976651+0000","last_fullsized":"2024-09-17T05:36:26.976651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4749,"log_dups_size":0,"ondisk_log_size":4749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294034242,"num_objects":76,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":70943,"num_write_kb":889582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.940713+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:35:25.940713+0000","last_peered":"2024-09-17T05:35:25.940713+0000","last_clean":"2024-09-17T05:35:25.940713+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:35:25.940713+0000","last_undegraded":"2024-09-17T05:35:25.940713+0000","last_fullsized":"2024-09-17T05:35:25.940713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.201104+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:35:26.201104+0000","last_peered":"2024-09-17T05:35:26.201104+0000","last_clean":"2024-09-17T05:35:26.201104+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:35:26.201104+0000","last_undegraded":"2024-09-17T05:35:26.201104+0000","last_fullsized":"2024-09-17T05:35:26.201104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":135,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187694+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:35:26.187694+0000","last_peered":"2024-09-17T05:35:26.187694+0000","last_clean":"2024-09-17T05:35:26.187694+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:35:26.187694+0000","last_undegraded":"2024-09-17T05:35:26.187694+0000","last_fullsized":"2024-09-17T05:35:26.187694+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'5318","reported_seq":5367,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.734064+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:36:24.734064+0000","last_peered":"2024-09-17T05:36:24.734064+0000","last_clean":"2024-09-17T05:36:24.734064+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:36:24.734064+0000","last_undegraded":"2024-09-17T05:36:24.734064+0000","last_fullsized":"2024-09-17T05:36:24.734064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5318,"log_dups_size":0,"ondisk_log_size":5318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398468612,"num_objects":100,"num_object_clones":0,"num_object_copies":400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":79949,"num_write_kb":1015697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065270+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:35:26.065270+0000","last_peered":"2024-09-17T05:35:26.065270+0000","last_clean":"2024-09-17T05:35:26.065270+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:35:26.065270+0000","last_undegraded":"2024-09-17T05:35:26.065270+0000","last_fullsized":"2024-09-17T05:35:26.065270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"30'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.981638+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:35:25.981638+0000","last_peered":"2024-09-17T05:35:25.981638+0000","last_clean":"2024-09-17T05:35:25.981638+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:35:25.981638+0000","last_undegraded":"2024-09-17T05:35:25.981638+0000","last_fullsized":"2024-09-17T05:35:25.981638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"31'5485","reported_seq":5533,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.259874+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:36:25.259874+0000","last_peered":"2024-09-17T05:36:25.259874+0000","last_clean":"2024-09-17T05:36:25.259874+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:36:25.259874+0000","last_undegraded":"2024-09-17T05:36:25.259874+0000","last_fullsized":"2024-09-17T05:36:25.259874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373295017,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82225,"num_write_kb":1025442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943229+0000","last_change":"2024-09-17T05:27:26.659915+0000","last_active":"2024-09-17T05:35:25.943229+0000","last_peered":"2024-09-17T05:35:25.943229+0000","last_clean":"2024-09-17T05:35:25.943229+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:35:25.943229+0000","last_undegraded":"2024-09-17T05:35:25.943229+0000","last_fullsized":"2024-09-17T05:35:25.943229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:55.237711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"30'20","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.072706+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:35:32.072706+0000","last_peered":"2024-09-17T05:35:32.072706+0000","last_clean":"2024-09-17T05:35:32.072706+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:35:32.072706+0000","last_undegraded":"2024-09-17T05:35:32.072706+0000","last_fullsized":"2024-09-17T05:35:32.072706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":19,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"31'5002","reported_seq":5050,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.107012+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:36:27.107012+0000","last_peered":"2024-09-17T05:36:27.107012+0000","last_clean":"2024-09-17T05:36:27.107012+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:36:27.107012+0000","last_undegraded":"2024-09-17T05:36:27.107012+0000","last_fullsized":"2024-09-17T05:36:27.107012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314598708,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":74882,"num_write_kb":933970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.946846+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:35:25.946846+0000","last_peered":"2024-09-17T05:35:25.946846+0000","last_clean":"2024-09-17T05:35:25.946846+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:35:25.946846+0000","last_undegraded":"2024-09-17T05:35:25.946846+0000","last_fullsized":"2024-09-17T05:35:25.946846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:31.863905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'27","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.808293+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:35:50.808293+0000","last_peered":"2024-09-17T05:35:50.808293+0000","last_clean":"2024-09-17T05:35:50.808293+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:35:50.808293+0000","last_undegraded":"2024-09-17T05:35:50.808293+0000","last_fullsized":"2024-09-17T05:35:50.808293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":26,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"31'5083","reported_seq":5128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:20.292547+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:36:20.292547+0000","last_peered":"2024-09-17T05:36:20.292547+0000","last_clean":"2024-09-17T05:36:20.292547+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:36:20.292547+0000","last_undegraded":"2024-09-17T05:36:20.292547+0000","last_fullsized":"2024-09-17T05:36:20.292547+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394284666,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76376,"num_write_kb":955028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936873+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:35:25.936873+0000","last_peered":"2024-09-17T05:35:25.936873+0000","last_clean":"2024-09-17T05:35:25.936873+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:35:25.936873+0000","last_undegraded":"2024-09-17T05:35:25.936873+0000","last_fullsized":"2024-09-17T05:35:25.936873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"31'8","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.592840+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:35:34.592840+0000","last_peered":"2024-09-17T05:35:34.592840+0000","last_clean":"2024-09-17T05:35:34.592840+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:35:34.592840+0000","last_undegraded":"2024-09-17T05:35:34.592840+0000","last_fullsized":"2024-09-17T05:35:34.592840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:36.584161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"31'5113","reported_seq":5158,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.336370+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:36:26.336370+0000","last_peered":"2024-09-17T05:36:26.336370+0000","last_clean":"2024-09-17T05:36:26.336370+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:36:26.336370+0000","last_undegraded":"2024-09-17T05:36:26.336370+0000","last_fullsized":"2024-09-17T05:36:26.336370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344012237,"num_objects":86,"num_object_clones":0,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76601,"num_write_kb":970760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.073783+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:35:26.073783+0000","last_peered":"2024-09-17T05:35:26.073783+0000","last_clean":"2024-09-17T05:35:26.073783+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:35:26.073783+0000","last_undegraded":"2024-09-17T05:35:26.073783+0000","last_fullsized":"2024-09-17T05:35:26.073783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"30'19","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.833437+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:35:50.833437+0000","last_peered":"2024-09-17T05:35:50.833437+0000","last_clean":"2024-09-17T05:35:50.833437+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:35:50.833437+0000","last_undegraded":"2024-09-17T05:35:50.833437+0000","last_fullsized":"2024-09-17T05:35:50.833437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":40,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"31'5738","reported_seq":5789,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.859102+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:36:26.859102+0000","last_peered":"2024-09-17T05:36:26.859102+0000","last_clean":"2024-09-17T05:36:26.859102+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:36:26.859102+0000","last_undegraded":"2024-09-17T05:36:26.859102+0000","last_fullsized":"2024-09-17T05:36:26.859102+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331603745,"num_objects":90,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":85828,"num_write_kb":1086559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187433+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:35:26.187433+0000","last_peered":"2024-09-17T05:35:26.187433+0000","last_clean":"2024-09-17T05:35:26.187433+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:35:26.187433+0000","last_undegraded":"2024-09-17T05:35:26.187433+0000","last_fullsized":"2024-09-17T05:35:26.187433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:30:49.975342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"30'13","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943104+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:35:25.943104+0000","last_peered":"2024-09-17T05:35:25.943104+0000","last_clean":"2024-09-17T05:35:25.943104+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:35:25.943104+0000","last_undegraded":"2024-09-17T05:35:25.943104+0000","last_fullsized":"2024-09-17T05:35:25.943104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:52.777737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1689,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":13,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"31'5839","reported_seq":5887,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.046398+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:36:27.046398+0000","last_peered":"2024-09-17T05:36:27.046398+0000","last_clean":"2024-09-17T05:36:27.046398+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:36:27.046398+0000","last_undegraded":"2024-09-17T05:36:27.046398+0000","last_fullsized":"2024-09-17T05:36:27.046398+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339798246,"num_objects":87,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87347,"num_write_kb":1070021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.200625+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:35:26.200625+0000","last_peered":"2024-09-17T05:35:26.200625+0000","last_clean":"2024-09-17T05:35:26.200625+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:35:26.200625+0000","last_undegraded":"2024-09-17T05:35:26.200625+0000","last_fullsized":"2024-09-17T05:35:26.200625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"30'6","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335841+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:35:26.335841+0000","last_peered":"2024-09-17T05:35:26.335841+0000","last_clean":"2024-09-17T05:35:26.335841+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:35:26.335841+0000","last_undegraded":"2024-09-17T05:35:26.335841+0000","last_fullsized":"2024-09-17T05:35:26.335841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'5863","reported_seq":5910,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.484944+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:36:27.484944+0000","last_peered":"2024-09-17T05:36:27.484944+0000","last_clean":"2024-09-17T05:36:27.484944+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:36:27.484944+0000","last_undegraded":"2024-09-17T05:36:27.484944+0000","last_fullsized":"2024-09-17T05:36:27.484944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5863,"log_dups_size":0,"ondisk_log_size":5863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318999243,"num_objects":82,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87526,"num_write_kb":1085839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"28'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970702+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:35:25.970702+0000","last_peered":"2024-09-17T05:35:25.970702+0000","last_clean":"2024-09-17T05:35:25.970702+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:35:25.970702+0000","last_undegraded":"2024-09-17T05:35:25.970702+0000","last_fullsized":"2024-09-17T05:35:25.970702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"30'87","reported_seq":134,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.501205+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:35:31.501205+0000","last_peered":"2024-09-17T05:35:31.501205+0000","last_clean":"2024-09-17T05:35:31.501205+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:35:31.501205+0000","last_undegraded":"2024-09-17T05:35:31.501205+0000","last_fullsized":"2024-09-17T05:35:31.501205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":8218,"num_write":88,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"31'5132","reported_seq":5178,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.833860+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:36:22.833860+0000","last_peered":"2024-09-17T05:36:22.833860+0000","last_clean":"2024-09-17T05:36:22.833860+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:36:22.833860+0000","last_undegraded":"2024-09-17T05:36:22.833860+0000","last_fullsized":"2024-09-17T05:36:22.833860+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394264576,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":76895,"num_write_kb":960359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.945356+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:35:25.945356+0000","last_peered":"2024-09-17T05:35:25.945356+0000","last_clean":"2024-09-17T05:35:25.945356+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:35:25.945356+0000","last_undegraded":"2024-09-17T05:35:25.945356+0000","last_fullsized":"2024-09-17T05:35:25.945356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"31'213","reported_seq":258,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:15.763813+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:36:15.763813+0000","last_peered":"2024-09-17T05:36:15.763813+0000","last_clean":"2024-09-17T05:36:15.763813+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:36:15.763813+0000","last_undegraded":"2024-09-17T05:36:15.763813+0000","last_fullsized":"2024-09-17T05:36:15.763813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":212,"num_write_kb":4244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"31'5323","reported_seq":5372,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.196028+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:36:25.196028+0000","last_peered":"2024-09-17T05:36:25.196028+0000","last_clean":"2024-09-17T05:36:25.196028+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:36:25.196028+0000","last_undegraded":"2024-09-17T05:36:25.196028+0000","last_fullsized":"2024-09-17T05:36:25.196028+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390153058,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79495,"num_write_kb":1008039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"28'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997367+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:35:25.997367+0000","last_peered":"2024-09-17T05:35:25.997367+0000","last_clean":"2024-09-17T05:35:25.997367+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:35:25.997367+0000","last_undegraded":"2024-09-17T05:35:25.997367+0000","last_fullsized":"2024-09-17T05:35:25.997367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"31'10","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.839506+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:35:50.839506+0000","last_peered":"2024-09-17T05:35:50.839506+0000","last_clean":"2024-09-17T05:35:50.839506+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:35:50.839506+0000","last_undegraded":"2024-09-17T05:35:50.839506+0000","last_fullsized":"2024-09-17T05:35:50.839506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"31'5584","reported_seq":5628,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.389841+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:36:25.389841+0000","last_peered":"2024-09-17T05:36:25.389841+0000","last_clean":"2024-09-17T05:36:25.389841+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:36:25.389841+0000","last_undegraded":"2024-09-17T05:36:25.389841+0000","last_fullsized":"2024-09-17T05:36:25.389841+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390177777,"num_objects":103,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":15,"num_read_kb":0,"num_write":83378,"num_write_kb":1022097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952372+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:35:25.952372+0000","last_peered":"2024-09-17T05:35:25.952372+0000","last_clean":"2024-09-17T05:35:25.952372+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:35:25.952372+0000","last_undegraded":"2024-09-17T05:35:25.952372+0000","last_fullsized":"2024-09-17T05:35:25.952372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"30'3","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943930+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:35:25.943930+0000","last_peered":"2024-09-17T05:35:25.943930+0000","last_clean":"2024-09-17T05:35:25.943930+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:35:25.943930+0000","last_undegraded":"2024-09-17T05:35:25.943930+0000","last_fullsized":"2024-09-17T05:35:25.943930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:52.191388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"31'5306","reported_seq":5349,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.902604+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:36:24.902604+0000","last_peered":"2024-09-17T05:36:24.902604+0000","last_clean":"2024-09-17T05:36:24.902604+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:36:24.902604+0000","last_undegraded":"2024-09-17T05:36:24.902604+0000","last_fullsized":"2024-09-17T05:36:24.902604+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385944448,"num_objects":99,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":13,"num_read_kb":0,"num_write":79626,"num_write_kb":1002963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"28'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936737+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:35:25.936737+0000","last_peered":"2024-09-17T05:35:25.936737+0000","last_clean":"2024-09-17T05:35:25.936737+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:35:25.936737+0000","last_undegraded":"2024-09-17T05:35:25.936737+0000","last_fullsized":"2024-09-17T05:35:25.936737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"31'11","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.972204+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:35:25.972204+0000","last_peered":"2024-09-17T05:35:25.972204+0000","last_clean":"2024-09-17T05:35:25.972204+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:35:25.972204+0000","last_undegraded":"2024-09-17T05:35:25.972204+0000","last_fullsized":"2024-09-17T05:35:25.972204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'5506","reported_seq":5553,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.983518+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:36:24.983518+0000","last_peered":"2024-09-17T05:36:24.983518+0000","last_clean":"2024-09-17T05:36:24.983518+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:36:24.983518+0000","last_undegraded":"2024-09-17T05:36:24.983518+0000","last_fullsized":"2024-09-17T05:36:24.983518+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406877841,"num_objects":103,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82610,"num_write_kb":1020969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6379","reported_seq":6427,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.313591+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:36:25.313591+0000","last_peered":"2024-09-17T05:36:25.313591+0000","last_clean":"2024-09-17T05:36:25.313591+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:36:25.313591+0000","last_undegraded":"2024-09-17T05:36:25.313591+0000","last_fullsized":"2024-09-17T05:36:25.313591+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452996234,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":95842,"num_write_kb":1202164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953061+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:35:25.953061+0000","last_peered":"2024-09-17T05:35:25.953061+0000","last_clean":"2024-09-17T05:35:25.953061+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:35:25.953061+0000","last_undegraded":"2024-09-17T05:35:25.953061+0000","last_fullsized":"2024-09-17T05:35:25.953061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"30'27","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.703337+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:35:31.703337+0000","last_peered":"2024-09-17T05:35:31.703337+0000","last_clean":"2024-09-17T05:35:31.703337+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:35:31.703337+0000","last_undegraded":"2024-09-17T05:35:31.703337+0000","last_fullsized":"2024-09-17T05:35:31.703337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":24,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"31'5525","reported_seq":5577,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:23.122693+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:36:23.122693+0000","last_peered":"2024-09-17T05:36:23.122693+0000","last_clean":"2024-09-17T05:36:23.122693+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:36:23.122693+0000","last_undegraded":"2024-09-17T05:36:23.122693+0000","last_fullsized":"2024-09-17T05:36:23.122693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5525,"log_dups_size":0,"ondisk_log_size":5525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427902156,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":82949,"num_write_kb":1044536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"28'14","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949598+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:35:25.949598+0000","last_peered":"2024-09-17T05:35:25.949598+0000","last_clean":"2024-09-17T05:35:25.949598+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:35:25.949598+0000","last_undegraded":"2024-09-17T05:35:25.949598+0000","last_fullsized":"2024-09-17T05:35:25.949598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"30'5","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962325+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:35:25.962325+0000","last_peered":"2024-09-17T05:35:25.962325+0000","last_clean":"2024-09-17T05:35:25.962325+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:35:25.962325+0000","last_undegraded":"2024-09-17T05:35:25.962325+0000","last_fullsized":"2024-09-17T05:35:25.962325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"31'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.941350+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:35:30.941350+0000","last_peered":"2024-09-17T05:35:30.941350+0000","last_clean":"2024-09-17T05:35:30.941350+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:35:30.941350+0000","last_undegraded":"2024-09-17T05:35:30.941350+0000","last_fullsized":"2024-09-17T05:35:30.941350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4108,"num_write":17,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941766+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:35:25.941766+0000","last_peered":"2024-09-17T05:35:25.941766+0000","last_clean":"2024-09-17T05:35:25.941766+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:35:25.941766+0000","last_undegraded":"2024-09-17T05:35:25.941766+0000","last_fullsized":"2024-09-17T05:35:25.941766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4537","reported_seq":4575,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.435603+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:36:25.435603+0000","last_peered":"2024-09-17T05:36:25.435603+0000","last_clean":"2024-09-17T05:36:25.435603+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:36:25.435603+0000","last_undegraded":"2024-09-17T05:36:25.435603+0000","last_fullsized":"2024-09-17T05:36:25.435603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4537,"log_dups_size":0,"ondisk_log_size":4537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297797311,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":68041,"num_write_kb":833128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1234","reported_seq":1275,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.957097+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:35:25.957097+0000","last_peered":"2024-09-17T05:35:25.957097+0000","last_clean":"2024-09-17T05:35:25.957097+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:35:25.957097+0000","last_undegraded":"2024-09-17T05:35:25.957097+0000","last_fullsized":"2024-09-17T05:35:25.957097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1234,"log_dups_size":0,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4196027,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1229,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968584+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:35:25.968584+0000","last_peered":"2024-09-17T05:35:25.968584+0000","last_clean":"2024-09-17T05:35:25.968584+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:35:25.968584+0000","last_undegraded":"2024-09-17T05:35:25.968584+0000","last_fullsized":"2024-09-17T05:35:25.968584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'6124","reported_seq":6173,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.242706+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:36:19.242706+0000","last_peered":"2024-09-17T05:36:19.242706+0000","last_clean":"2024-09-17T05:36:19.242706+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:36:19.242706+0000","last_undegraded":"2024-09-17T05:36:19.242706+0000","last_fullsized":"2024-09-17T05:36:19.242706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6124,"log_dups_size":0,"ondisk_log_size":6124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465673452,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":92183,"num_write_kb":1140422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.935850+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:35:25.935850+0000","last_peered":"2024-09-17T05:35:25.935850+0000","last_clean":"2024-09-17T05:35:25.935850+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:35:25.935850+0000","last_undegraded":"2024-09-17T05:35:25.935850+0000","last_fullsized":"2024-09-17T05:35:25.935850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.960667+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:35:25.960667+0000","last_peered":"2024-09-17T05:35:25.960667+0000","last_clean":"2024-09-17T05:35:25.960667+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:35:25.960667+0000","last_undegraded":"2024-09-17T05:35:25.960667+0000","last_fullsized":"2024-09-17T05:35:25.960667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"28'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335365+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:35:26.335365+0000","last_peered":"2024-09-17T05:35:26.335365+0000","last_clean":"2024-09-17T05:35:26.335365+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:35:26.335365+0000","last_undegraded":"2024-09-17T05:35:26.335365+0000","last_fullsized":"2024-09-17T05:35:26.335365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.876152+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:35:55.876152+0000","last_peered":"2024-09-17T05:35:55.876152+0000","last_clean":"2024-09-17T05:35:55.876152+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:35:55.876152+0000","last_undegraded":"2024-09-17T05:35:55.876152+0000","last_fullsized":"2024-09-17T05:35:55.876152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559514+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:36:06.559514+0000","last_peered":"2024-09-17T05:36:06.559514+0000","last_clean":"2024-09-17T05:36:06.559514+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:36:06.559514+0000","last_undegraded":"2024-09-17T05:36:06.559514+0000","last_fullsized":"2024-09-17T05:36:06.559514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559546+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:36:06.559546+0000","last_peered":"2024-09-17T05:36:06.559546+0000","last_clean":"2024-09-17T05:36:06.559546+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:36:06.559546+0000","last_undegraded":"2024-09-17T05:36:06.559546+0000","last_fullsized":"2024-09-17T05:36:06.559546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"28'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.936653+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:35:25.936653+0000","last_peered":"2024-09-17T05:35:25.936653+0000","last_clean":"2024-09-17T05:35:25.936653+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:35:25.936653+0000","last_undegraded":"2024-09-17T05:35:25.936653+0000","last_fullsized":"2024-09-17T05:35:25.936653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"28'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.952118+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:35:25.952118+0000","last_peered":"2024-09-17T05:35:25.952118+0000","last_clean":"2024-09-17T05:35:25.952118+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:35:25.952118+0000","last_undegraded":"2024-09-17T05:35:25.952118+0000","last_fullsized":"2024-09-17T05:35:25.952118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"28'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.193551+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:35:26.193551+0000","last_peered":"2024-09-17T05:35:26.193551+0000","last_clean":"2024-09-17T05:35:26.193551+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:35:26.193551+0000","last_undegraded":"2024-09-17T05:35:26.193551+0000","last_fullsized":"2024-09-17T05:35:26.193551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"31'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:22.061407+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:36:22.061407+0000","last_peered":"2024-09-17T05:36:22.061407+0000","last_clean":"2024-09-17T05:36:22.061407+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:36:22.061407+0000","last_undegraded":"2024-09-17T05:36:22.061407+0000","last_fullsized":"2024-09-17T05:36:22.061407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968478+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:35:25.968478+0000","last_peered":"2024-09-17T05:35:25.968478+0000","last_clean":"2024-09-17T05:35:25.968478+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:35:25.968478+0000","last_undegraded":"2024-09-17T05:35:25.968478+0000","last_fullsized":"2024-09-17T05:35:25.968478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"28'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941600+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:35:25.941600+0000","last_peered":"2024-09-17T05:35:25.941600+0000","last_clean":"2024-09-17T05:35:25.941600+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:35:25.941600+0000","last_undegraded":"2024-09-17T05:35:25.941600+0000","last_fullsized":"2024-09-17T05:35:25.941600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"31'7","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943098+0000","last_change":"2024-09-17T05:27:26.660260+0000","last_active":"2024-09-17T05:35:25.943098+0000","last_peered":"2024-09-17T05:35:25.943098+0000","last_clean":"2024-09-17T05:35:25.943098+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:35:25.943098+0000","last_undegraded":"2024-09-17T05:35:25.943098+0000","last_fullsized":"2024-09-17T05:35:25.943098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.065935+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:35:26.065935+0000","last_peered":"2024-09-17T05:35:26.065935+0000","last_clean":"2024-09-17T05:35:26.065935+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:35:26.065935+0000","last_undegraded":"2024-09-17T05:35:26.065935+0000","last_fullsized":"2024-09-17T05:35:26.065935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"28'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.948268+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:35:25.948268+0000","last_peered":"2024-09-17T05:35:25.948268+0000","last_clean":"2024-09-17T05:35:25.948268+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:35:25.948268+0000","last_undegraded":"2024-09-17T05:35:25.948268+0000","last_fullsized":"2024-09-17T05:35:25.948268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"28'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.168925+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:35:26.168925+0000","last_peered":"2024-09-17T05:35:26.168925+0000","last_clean":"2024-09-17T05:35:26.168925+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:35:26.168925+0000","last_undegraded":"2024-09-17T05:35:26.168925+0000","last_fullsized":"2024-09-17T05:35:26.168925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.193404+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:35:26.193404+0000","last_peered":"2024-09-17T05:35:26.193404+0000","last_clean":"2024-09-17T05:35:26.193404+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:35:26.193404+0000","last_undegraded":"2024-09-17T05:35:26.193404+0000","last_fullsized":"2024-09-17T05:35:26.193404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"28'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.951216+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:35:25.951216+0000","last_peered":"2024-09-17T05:35:25.951216+0000","last_clean":"2024-09-17T05:35:25.951216+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:35:25.951216+0000","last_undegraded":"2024-09-17T05:35:25.951216+0000","last_fullsized":"2024-09-17T05:35:25.951216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.970393+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:35:25.970393+0000","last_peered":"2024-09-17T05:35:25.970393+0000","last_clean":"2024-09-17T05:35:25.970393+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:35:25.970393+0000","last_undegraded":"2024-09-17T05:35:25.970393+0000","last_fullsized":"2024-09-17T05:35:25.970393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"28'8","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997235+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:35:25.997235+0000","last_peered":"2024-09-17T05:35:25.997235+0000","last_clean":"2024-09-17T05:35:25.997235+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:35:25.997235+0000","last_undegraded":"2024-09-17T05:35:25.997235+0000","last_fullsized":"2024-09-17T05:35:25.997235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"28'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.942921+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:35:25.942921+0000","last_peered":"2024-09-17T05:35:25.942921+0000","last_clean":"2024-09-17T05:35:25.942921+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:35:25.942921+0000","last_undegraded":"2024-09-17T05:35:25.942921+0000","last_fullsized":"2024-09-17T05:35:25.942921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"28'11","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943835+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:35:25.943835+0000","last_peered":"2024-09-17T05:35:25.943835+0000","last_clean":"2024-09-17T05:35:25.943835+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:35:25.943835+0000","last_undegraded":"2024-09-17T05:35:25.943835+0000","last_fullsized":"2024-09-17T05:35:25.943835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943813+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:35:25.943813+0000","last_peered":"2024-09-17T05:35:25.943813+0000","last_clean":"2024-09-17T05:35:25.943813+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:35:25.943813+0000","last_undegraded":"2024-09-17T05:35:25.943813+0000","last_fullsized":"2024-09-17T05:35:25.943813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"31'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.976708+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:35:25.976708+0000","last_peered":"2024-09-17T05:35:25.976708+0000","last_clean":"2024-09-17T05:35:25.976708+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:35:25.976708+0000","last_undegraded":"2024-09-17T05:35:25.976708+0000","last_fullsized":"2024-09-17T05:35:25.976708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"28'3","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949226+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:35:25.949226+0000","last_peered":"2024-09-17T05:35:25.949226+0000","last_clean":"2024-09-17T05:35:25.949226+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:35:25.949226+0000","last_undegraded":"2024-09-17T05:35:25.949226+0000","last_fullsized":"2024-09-17T05:35:25.949226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"31'4","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943027+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:35:25.943027+0000","last_peered":"2024-09-17T05:35:25.943027+0000","last_clean":"2024-09-17T05:35:25.943027+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:35:25.943027+0000","last_undegraded":"2024-09-17T05:35:25.943027+0000","last_fullsized":"2024-09-17T05:35:25.943027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:08.554808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"28'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947757+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:35:25.947757+0000","last_peered":"2024-09-17T05:35:25.947757+0000","last_clean":"2024-09-17T05:35:25.947757+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:35:25.947757+0000","last_undegraded":"2024-09-17T05:35:25.947757+0000","last_fullsized":"2024-09-17T05:35:25.947757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"28'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.943661+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:35:25.943661+0000","last_peered":"2024-09-17T05:35:25.943661+0000","last_clean":"2024-09-17T05:35:25.943661+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:35:25.943661+0000","last_undegraded":"2024-09-17T05:35:25.943661+0000","last_fullsized":"2024-09-17T05:35:25.943661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:21.741035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"28'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.935304+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:35:25.935304+0000","last_peered":"2024-09-17T05:35:25.935304+0000","last_clean":"2024-09-17T05:35:25.935304+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:35:25.935304+0000","last_undegraded":"2024-09-17T05:35:25.935304+0000","last_fullsized":"2024-09-17T05:35:25.935304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:40.792244+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:35:40.792244+0000","last_peered":"2024-09-17T05:35:40.792244+0000","last_clean":"2024-09-17T05:35:40.792244+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:35:40.792244+0000","last_undegraded":"2024-09-17T05:35:40.792244+0000","last_fullsized":"2024-09-17T05:35:40.792244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"28'2","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.969435+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:35:25.969435+0000","last_peered":"2024-09-17T05:35:25.969435+0000","last_clean":"2024-09-17T05:35:25.969435+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:35:25.969435+0000","last_undegraded":"2024-09-17T05:35:25.969435+0000","last_fullsized":"2024-09-17T05:35:25.969435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"28'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947901+0000","last_change":"2024-09-17T05:28:07.483393+0000","last_active":"2024-09-17T05:35:25.947901+0000","last_peered":"2024-09-17T05:35:25.947901+0000","last_clean":"2024-09-17T05:35:25.947901+0000","last_became_active":"2024-09-17T05:28:07.483003+0000","last_became_peered":"2024-09-17T05:28:07.483003+0000","last_unstale":"2024-09-17T05:35:25.947901+0000","last_undegraded":"2024-09-17T05:35:25.947901+0000","last_fullsized":"2024-09-17T05:35:25.947901+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":11805228873,"num_objects":3005,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":525,"num_read_kb":0,"num_write":2618859,"num_write_kb":32721427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12280737792,"data_stored":23555850240,"data_compressed":1201049330,"data_compressed_allocated":4969824256,"data_compressed_original":16244936704,"omap_allocated":0,"internal_metadata":0},"log_size":174780,"ondisk_log_size":174780,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":471,"ondisk_log_size":471,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":82155219,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":346,"num_read_kb":78280,"num_write":2707,"num_write_kb":82299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":33984512,"data_stored":164186953,"data_compressed":18806052,"data_compressed_allocated":21831680,"data_compressed_original":152186880,"omap_allocated":0,"internal_metadata":0},"log_size":2529,"ondisk_log_size":2529,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509550,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3211116,"kb_used_data":2047708,"kb_used_omap":1,"kb_used_meta":1163390,"kb_avail":91160724,"statfs":{"total":96636764160,"available":93348581376,"internally_reserved":0,"allocated":2096852992,"data_stored":4054768616,"data_compressed":208583112,"data_compressed_allocated":849747968,"data_compressed_original":2807865344,"omap_allocated":1593,"internal_metadata":1191311815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,0,0,0,0,0,0,30,93,11],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509552,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2199544,"kb_used_data":1403112,"kb_used_omap":1,"kb_used_meta":796414,"kb_avail":92172296,"statfs":{"total":96636764160,"available":94384431104,"internally_reserved":0,"allocated":1436786688,"data_stored":2758514067,"data_compressed":141474862,"data_compressed_allocated":581439488,"data_compressed_original":1903366144,"omap_allocated":1595,"internal_metadata":815528389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509552,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2990224,"kb_used_data":1907136,"kb_used_omap":1,"kb_used_meta":1083070,"kb_avail":91381616,"statfs":{"total":96636764160,"available":93574774784,"internally_reserved":0,"allocated":1952907264,"data_stored":3778817967,"data_compressed":194628104,"data_compressed_allocated":793247744,"data_compressed_original":2619359232,"omap_allocated":1595,"internal_metadata":1109064133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509553,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1602876,"kb_used_data":917612,"kb_used_omap":1,"kb_used_meta":685246,"kb_avail":92768964,"statfs":{"total":96636764160,"available":94995419136,"internally_reserved":0,"allocated":939634688,"data_stored":1833875455,"data_compressed":95381766,"data_compressed_allocated":384143360,"data_compressed_original":1278590976,"omap_allocated":1595,"internal_metadata":701692357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542254,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2975728,"kb_used_data":1898656,"kb_used_omap":1,"kb_used_meta":1077054,"kb_avail":91396112,"statfs":{"total":96636764160,"available":93589618688,"internally_reserved":0,"allocated":1944223744,"data_stored":3742764326,"data_compressed":190498180,"data_compressed_allocated":783704064,"data_compressed_original":2582458368,"omap_allocated":1590,"internal_metadata":1102903754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542255,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2517912,"kb_used_data":1445448,"kb_used_omap":1,"kb_used_meta":1072446,"kb_avail":91853928,"statfs":{"total":96636764160,"available":94058422272,"internally_reserved":0,"allocated":1480138752,"data_stored":2846927826,"data_compressed":145986734,"data_compressed_allocated":601456640,"data_compressed_original":1968443392,"omap_allocated":1594,"internal_metadata":1098185158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2478492,"kb_used_data":1631372,"kb_used_omap":1,"kb_used_meta":847102,"kb_avail":91893348,"statfs":{"total":96636764160,"available":94098788352,"internally_reserved":0,"allocated":1670524928,"data_stored":3211504529,"data_compressed":163583899,"data_compressed_allocated":673783808,"data_compressed_original":2214989824,"omap_allocated":1594,"internal_metadata":867432902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2167944,"kb_used_data":1320952,"kb_used_omap":1,"kb_used_meta":846974,"kb_avail":92203896,"statfs":{"total":96636764160,"available":94416789504,"internally_reserved":0,"allocated":1352654848,"data_stored":2583291230,"data_compressed":133191837,"data_compressed_allocated":544722944,"data_compressed_original":1775570944,"omap_allocated":1595,"internal_metadata":867301829},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5005312,"data_stored":27563310,"data_compressed":3259739,"data_compressed_allocated":3784704,"data_compressed_original":26361856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":23225030,"data_compressed":2717358,"data_compressed_allocated":3166208,"data_compressed_original":22245376,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":12583618,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":37751063,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":12585152,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6688768,"data_stored":25247853,"data_compressed":2684105,"data_compressed_allocated":3153920,"data_compressed_original":21733376,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":12585217,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2154496,"data_stored":12645710,"data_compressed":1495438,"data_compressed_allocated":1753088,"data_compressed_original":12271616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":844763136,"data_stored":1632104448,"data_compressed":83506478,"data_compressed_allocated":344780800,"data_compressed_original":1132122112,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1789231104,"data_stored":3444797440,"data_compressed":175919946,"data_compressed_allocated":726069248,"data_compressed_original":2381635584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1422041088,"data_stored":2724814848,"data_compressed":138920823,"data_compressed_allocated":574853120,"data_compressed_original":1877626880,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2066087936,"data_stored":3956056064,"data_compressed":201398317,"data_compressed_allocated":834166784,"data_compressed_original":2724134912,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1544409088,"data_stored":2963894272,"data_compressed":151060241,"data_compressed_allocated":624812032,"data_compressed_original":2044297216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1331101696,"data_stored":2527948800,"data_compressed":128951423,"data_compressed_allocated":535752704,"data_compressed_original":1732599808,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1368395776,"data_stored":2639163392,"data_compressed":135161769,"data_compressed_allocated":558776320,"data_compressed_original":1829543936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1914707968,"data_stored":3667070976,"data_compressed":186130333,"data_compressed_allocated":770613248,"data_compressed_original":2522976256,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:36:31.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:36:31.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.368+0000 7fb0bf87e640 1 -- 172.21.15.117:0/2931798157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 msgr2=0x7fb0a4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.368+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/2931798157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0a4003e50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800c610 tx=0x7fb0a4005830 comp rx=0 tx=0).stop 2024-09-17T05:36:31.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.368+0000 7fb0bf87e640 1 -- 172.21.15.117:0/2931798157 shutdown_connections 2024-09-17T05:36:31.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.368+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/2931798157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0a4003e50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.368+0000 7fb0bf87e640 1 -- 172.21.15.117:0/2931798157 >> 172.21.15.117:0/2931798157 conn(0x7fb0b806e020 msgr2=0x7fb0b806eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.371+0000 7fb0bf87e640 1 -- 172.21.15.117:0/2931798157 shutdown_connections 2024-09-17T05:36:31.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.371+0000 7fb0bf87e640 1 -- 172.21.15.117:0/2931798157 wait complete. 2024-09-17T05:36:31.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.374+0000 7fb0bf87e640 1 Processor -- start 2024-09-17T05:36:31.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.374+0000 7fb0bf87e640 1 -- start start 2024-09-17T05:36:31.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.376+0000 7fb0bf87e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0b81f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:31.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.376+0000 7fb0bd61a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0b81f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:31.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bd61a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0b81f1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60518/0 (socket says 172.21.15.117:60518) 2024-09-17T05:36:31.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bd61a640 1 -- 172.21.15.117:0/3118822928 learned_addr learned my addr 172.21.15.117:0/3118822928 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:31.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0b815c340 0x7fb0b815d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:31.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0b8152e00 0x7fb0b8148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:31.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bce19640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0b815c340 0x7fb0b815d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:31.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b80c7120 con 0x7fb0a4003a00 2024-09-17T05:36:31.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.377+0000 7fb0bde1b640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0b8152e00 0x7fb0b8148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:31.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b815bc30 con 0x7fb0b815c340 2024-09-17T05:36:31.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b80c72a0 con 0x7fb0b8152e00 2024-09-17T05:36:31.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bd61a640 1 -- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0b8152e00 msgr2=0x7fb0b8148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bd61a640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0b8152e00 0x7fb0b8148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bd61a640 1 -- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0b815c340 msgr2=0x7fb0b815d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bd61a640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0b815c340 0x7fb0b815d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.378+0000 7fb0bd61a640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b81f2480 con 0x7fb0a4003a00 2024-09-17T05:36:31.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.379+0000 7fb0bd61a640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0b81f1f40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb0a802b360 tx=0x7fb0a800c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:31.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.380+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a802b8b0 con 0x7fb0a4003a00 2024-09-17T05:36:31.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.380+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0a802ba50 con 0x7fb0a4003a00 2024-09-17T05:36:31.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.380+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a802be60 con 0x7fb0a4003a00 2024-09-17T05:36:31.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.381+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b814b2d0 con 0x7fb0a4003a00 2024-09-17T05:36:31.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.382+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b814bd70 con 0x7fb0a4003a00 2024-09-17T05:36:31.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.383+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b81555c0 con 0x7fb0a4003a00 2024-09-17T05:36:31.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.384+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb0a80035f0 con 0x7fb0a4003a00 2024-09-17T05:36:31.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.384+0000 7fb0adffb640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb09c078c60 0x7fb09c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:31.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.385+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fb0a80bb6c0 con 0x7fb0a4003a00 2024-09-17T05:36:31.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.385+0000 7fb0bce19640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb09c078c60 0x7fb09c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:31.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.385+0000 7fb0bce19640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb09c078c60 0x7fb09c07b170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8151150 tx=0x7fb0b4005d60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:31.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.390+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb0b80c74f0 con 0x7fb0a4003a00 2024-09-17T05:36:31.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.748+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0b80cdf40 con 0x7fb0a4003a00 2024-09-17T05:36:31.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.757+0000 7fb0adffb640 1 -- 172.21.15.117:0/3118822928 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+19292 (secure 0 0 0) 0x7fb0a8032e50 con 0x7fb0a4003a00 2024-09-17T05:36:31.761 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:36:31.762 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":31,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:35:25.910864+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3f","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6839/2500745218":"2024-09-18T05:35:25.910809+0000","172.21.15.119:6838/2500745218":"2024-09-18T05:35:25.910809+0000","172.21.15.119:6835/2224210538":"2024-09-18T05:33:15.875991+0000","172.21.15.119:6834/2224210538":"2024-09-18T05:33:15.875991+0000","172.21.15.117:6835/235108685":"2024-09-18T05:31:45.857073+0000","172.21.15.117:6834/235108685":"2024-09-18T05:31:45.857073+0000","172.21.15.117:6837/242246295":"2024-09-18T05:29:10.830912+0000","172.21.15.117:6836/242246295":"2024-09-18T05:29:10.830912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb09c078c60 msgr2=0x7fb09c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb09c078c60 0x7fb09c07b170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8151150 tx=0x7fb0b4005d60 comp rx=0 tx=0).stop 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 msgr2=0x7fb0b81f1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0b81f1f40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb0a802b360 tx=0x7fb0a800c300 comp rx=0 tx=0).stop 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 shutdown_connections 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fb09c078c60 0x7fb09c07b170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0b8152e00 0x7fb0b8148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0b815c340 0x7fb0b815d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 --2- 172.21.15.117:0/3118822928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a4003a00 0x7fb0b81f1f40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:31.763 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 >> 172.21.15.117:0/3118822928 conn(0x7fb0b806e020 msgr2=0x7fb0b80c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:31.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.761+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 shutdown_connections 2024-09-17T05:36:31.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:31.762+0000 7fb0bf87e640 1 -- 172.21.15.117:0/3118822928 wait complete. 2024-09-17T05:36:32.780 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T05:36:32.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:32.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.965+0000 7fa5eee07640 1 -- 172.21.15.117:0/3313630603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 msgr2=0x7fa5e80d0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:32.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.965+0000 7fa5eee07640 1 --2- 172.21.15.117:0/3313630603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 0x7fa5e80d0c20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009830 tx=0x7fa5d802a390 comp rx=0 tx=0).stop 2024-09-17T05:36:32.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.968+0000 7fa5eee07640 1 -- 172.21.15.117:0/3313630603 shutdown_connections 2024-09-17T05:36:32.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.968+0000 7fa5eee07640 1 --2- 172.21.15.117:0/3313630603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 0x7fa5e80d0c20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:32.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.968+0000 7fa5eee07640 1 --2- 172.21.15.117:0/3313630603 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa5e80cbd80 0x7fa5e8155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:32.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.968+0000 7fa5eee07640 1 -- 172.21.15.117:0/3313630603 >> 172.21.15.117:0/3313630603 conn(0x7fa5e806e9b0 msgr2=0x7fa5e806edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:32.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.971+0000 7fa5eee07640 1 -- 172.21.15.117:0/3313630603 shutdown_connections 2024-09-17T05:36:32.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.974+0000 7fa5eee07640 1 -- 172.21.15.117:0/3313630603 wait complete. 2024-09-17T05:36:32.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.977+0000 7fa5eee07640 1 Processor -- start 2024-09-17T05:36:32.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.977+0000 7fa5eee07640 1 -- start start 2024-09-17T05:36:32.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.979+0000 7fa5eee07640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5e80c6330 0x7fa5e81e6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:32.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5eee07640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa5e80cbd80 0x7fa5e82024b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:32.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5eee07640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 0x7fa5e81f5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:32.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5ecba3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5e80c6330 0x7fa5e81e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:32.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5ecba3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5e80c6330 0x7fa5e81e6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:37042/0 (socket says 172.21.15.117:37042) 2024-09-17T05:36:32.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5e7fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa5e80cbd80 0x7fa5e82024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:32.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5e7fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa5e80cbd80 0x7fa5e82024b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:58692/0 (socket says 172.21.15.117:58692) 2024-09-17T05:36:32.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5ecba3640 1 -- 172.21.15.117:0/747052383 learned_addr learned my addr 172.21.15.117:0/747052383 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:32.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e815f490 con 0x7fa5e80ccb70 2024-09-17T05:36:32.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.980+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e80cc3a0 con 0x7fa5e80c6330 2024-09-17T05:36:32.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.981+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e80c5930 con 0x7fa5e80cbd80 2024-09-17T05:36:32.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.983+0000 7fa5ed3a4640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 0x7fa5e81f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:32.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.984+0000 7fa5ed3a4640 1 -- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa5e80cbd80 msgr2=0x7fa5e82024b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:32.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.984+0000 7fa5ed3a4640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa5e80cbd80 0x7fa5e82024b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:32.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.984+0000 7fa5ed3a4640 1 -- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5e80c6330 msgr2=0x7fa5e81e6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:32.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.984+0000 7fa5ed3a4640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5e80c6330 0x7fa5e81e6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:32.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.984+0000 7fa5ed3a4640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e82029f0 con 0x7fa5e80ccb70 2024-09-17T05:36:32.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.984+0000 7fa5ed3a4640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 0x7fa5e81f5020 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c770 tx=0x7fa5e000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:32.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.990+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0017070 con 0x7fa5e80ccb70 2024-09-17T05:36:32.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.991+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e80cff30 con 0x7fa5e80ccb70 2024-09-17T05:36:32.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.991+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e80d04b0 con 0x7fa5e80ccb70 2024-09-17T05:36:32.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.992+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5e0005070 con 0x7fa5e80ccb70 2024-09-17T05:36:32.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.992+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0005480 con 0x7fa5e80ccb70 2024-09-17T05:36:32.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.992+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5e8202060 con 0x7fa5e80ccb70 2024-09-17T05:36:32.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.995+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa5e001a070 con 0x7fa5e80ccb70 2024-09-17T05:36:32.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.996+0000 7fa5e57fa640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa5d4078c60 0x7fa5d407b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:32.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.996+0000 7fa5ecba3640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa5d4078c60 0x7fa5d407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:32.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.996+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fa5e009b390 con 0x7fa5e80ccb70 2024-09-17T05:36:32.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.997+0000 7fa5e57fa640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7fa5d40827d0 0x7fa5d4084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:32.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.997+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5d4085370 con 0x7fa5d40827d0 2024-09-17T05:36:32.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.997+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa5e009b780 con 0x7fa5e80ccb70 2024-09-17T05:36:32.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.997+0000 7fa5e7fff640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7fa5d40827d0 0x7fa5d4084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:32.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:32.998+0000 7fa5e7fff640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7fa5d40827d0 0x7fa5d4084ca0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:33.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.000+0000 7fa5ecba3640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa5d4078c60 0x7fa5d407b170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0058c0 tx=0x7fa5dc00a4e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:33.005 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.004+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== osd.0 v2:172.21.15.117:6816/2282130530 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5d4085370 con 0x7fa5d40827d0 2024-09-17T05:36:33.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.028+0000 7fa5eee07640 1 -- 172.21.15.117:0/747052383 --> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa5e81ff610 con 0x7fa5d40827d0 2024-09-17T05:36:33.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.037+0000 7fa5e57fa640 1 -- 172.21.15.117:0/747052383 <== osd.0 v2:172.21.15.117:6816/2282130530 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa5e81ff610 con 0x7fa5d40827d0 2024-09-17T05:36:33.038 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:33.038 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:33.038 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.038+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7fa5d40827d0 msgr2=0x7fa5d4084ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.038+0000 7fa5ceffd640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:6816/2282130530,v1:172.21.15.117:6817/2282130530] conn(0x7fa5d40827d0 0x7fa5d4084ca0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.039+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa5d4078c60 msgr2=0x7fa5d407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.039+0000 7fa5ceffd640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fa5d4078c60 0x7fa5d407b170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0058c0 tx=0x7fa5dc00a4e0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.039+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 msgr2=0x7fa5e81f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.039+0000 7fa5ceffd640 1 --2- 172.21.15.117:0/747052383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5e80ccb70 0x7fa5e81f5020 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000c770 tx=0x7fa5e000cc30 comp rx=0 tx=0).stop 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.040+0000 7fa5ed3a4640 1 -- 172.21.15.117:0/747052383 reap_dead start 2024-09-17T05:36:33.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.040+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 shutdown_connections 2024-09-17T05:36:33.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.040+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 >> 172.21.15.117:0/747052383 conn(0x7fa5e806e9b0 msgr2=0x7fa5e81e7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:33.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.045+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 shutdown_connections 2024-09-17T05:36:33.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.045+0000 7fa5ceffd640 1 -- 172.21.15.117:0/747052383 wait complete. 2024-09-17T05:36:33.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T05:36:33.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.304+0000 7f6ae969b640 1 -- 172.21.15.117:0/2367068943 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c7120 msgr2=0x7f6ae40cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.304+0000 7f6ae969b640 1 --2- 172.21.15.117:0/2367068943 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c7120 0x7f6ae40cbb00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4005fd0 tx=0x7f6ad402a5f0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.309+0000 7f6ae969b640 1 -- 172.21.15.117:0/2367068943 shutdown_connections 2024-09-17T05:36:33.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.309+0000 7f6ae969b640 1 --2- 172.21.15.117:0/2367068943 >> v2:172.21.15.117:3300/0 conn(0x7f6ae40c5ad0 0x7f6ae40c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.309+0000 7f6ae969b640 1 --2- 172.21.15.117:0/2367068943 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c7120 0x7f6ae40cbb00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.309+0000 7f6ae969b640 1 -- 172.21.15.117:0/2367068943 >> 172.21.15.117:0/2367068943 conn(0x7f6ae406e020 msgr2=0x7f6ae406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:33.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.311+0000 7f6ae969b640 1 -- 172.21.15.117:0/2367068943 shutdown_connections 2024-09-17T05:36:33.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.311+0000 7f6ae969b640 1 -- 172.21.15.117:0/2367068943 wait complete. 2024-09-17T05:36:33.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.315+0000 7f6ae969b640 1 Processor -- start 2024-09-17T05:36:33.316 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.315+0000 7f6ae969b640 1 -- start start 2024-09-17T05:36:33.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.317+0000 7f6ae969b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c5ad0 0x7f6ae40cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.317+0000 7f6ae969b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 0x7f6ae41570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.317+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c5ad0 0x7f6ae40cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.317+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c5ad0 0x7f6ae40cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.117:58708/0 (socket says 172.21.15.117:58708) 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ada7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 0x7f6ae41570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ada7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 0x7f6ae41570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60564/0 (socket says 172.21.15.117:60564) 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ae969b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ae415e870 0x7f6ae4155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ae969b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae415bbf0 con 0x7f6ae40c7120 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ae969b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae415b910 con 0x7f6ae415e870 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ae969b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae40cc040 con 0x7f6ae40c5ad0 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ada7fc640 1 -- 172.21.15.117:0/3816762462 learned_addr learned my addr 172.21.15.117:0/3816762462 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:33.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.318+0000 7f6ae37fe640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ae415e870 0x7f6ae4155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.319+0000 7f6ae37fe640 1 -- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c5ad0 msgr2=0x7f6ae40cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.319+0000 7f6ae37fe640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c5ad0 0x7f6ae40cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.319+0000 7f6ae37fe640 1 -- 172.21.15.117:0/3816762462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 msgr2=0x7f6ae41570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.319+0000 7f6ae37fe640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 0x7f6ae41570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.320+0000 7f6ae37fe640 1 -- 172.21.15.117:0/3816762462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae41f61a0 con 0x7f6ae415e870 2024-09-17T05:36:33.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.320+0000 7f6ada7fc640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 0x7f6ae41570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:36:33.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.320+0000 7f6ae37fe640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ae415e870 0x7f6ae4155cd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6ae408ec20 tx=0x7f6adc009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:33.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.321+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc019070 con 0x7f6ae415e870 2024-09-17T05:36:33.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.321+0000 7f6ae969b640 1 -- 172.21.15.117:0/3816762462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae41f6440 con 0x7f6ae415e870 2024-09-17T05:36:33.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.321+0000 7f6ae969b640 1 -- 172.21.15.117:0/3816762462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae40cb120 con 0x7f6ae415e870 2024-09-17T05:36:33.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.323+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae406af90 con 0x7f6ae415e870 2024-09-17T05:36:33.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.324+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6adc014440 con 0x7f6ae415e870 2024-09-17T05:36:33.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.324+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0148d0 con 0x7f6ae415e870 2024-09-17T05:36:33.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.327+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6adc01c070 con 0x7f6ae415e870 2024-09-17T05:36:33.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.328+0000 7f6adbfff640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6ac4078bc0 0x7f6ac407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.328+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f6adc09b280 con 0x7f6ae415e870 2024-09-17T05:36:33.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.331+0000 7f6ae2ffd640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6ac4078bc0 0x7f6ac407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.336+0000 7f6ae2ffd640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6ac4078bc0 0x7f6ac407b0d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4032000 tx=0x7f6ad4004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:33.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.338+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6ae406af90 con 0x7f6ae415e870 2024-09-17T05:36:33.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.703+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f6ae40d00a0 con 0x7f6ac4078bc0 2024-09-17T05:36:33.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.705+0000 7f6adbfff640 1 -- 172.21.15.117:0/3816762462 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6ae40d00a0 con 0x7f6ac4078bc0 2024-09-17T05:36:33.711 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6ac4078bc0 msgr2=0x7f6ac407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6ac4078bc0 0x7f6ac407b0d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4032000 tx=0x7f6ad4004730 comp rx=0 tx=0).stop 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ae415e870 msgr2=0x7f6ae4155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ae415e870 0x7f6ae4155cd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6ae408ec20 tx=0x7f6adc009ed0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 shutdown_connections 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f6ac4078bc0 0x7f6ac407b0d0 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4032000 tx=0x7f6ad4004730 comp rx=0 tx=0).stop 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ae415e870 0x7f6ae4155cd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae40c7120 0x7f6ae41570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 --2- 172.21.15.117:0/3816762462 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6ae40c5ad0 0x7f6ae40cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.715+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 >> 172.21.15.117:0/3816762462 conn(0x7f6ae406e020 msgr2=0x7f6ae415ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:33.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.716+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 shutdown_connections 2024-09-17T05:36:33.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.716+0000 7f6ad97fa640 1 -- 172.21.15.117:0/3816762462 wait complete. 2024-09-17T05:36:33.736 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T05:36:33.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:33.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.951+0000 7f4985cb4640 1 -- 172.21.15.117:0/2309849018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 msgr2=0x7f49801516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.951+0000 7f4985cb4640 1 --2- 172.21.15.117:0/2309849018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49801516d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4978005fd0 tx=0x7f497802b8e0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.962+0000 7f4985cb4640 1 -- 172.21.15.117:0/2309849018 shutdown_connections 2024-09-17T05:36:33.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.962+0000 7f4985cb4640 1 --2- 172.21.15.117:0/2309849018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49801516d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.962+0000 7f4985cb4640 1 -- 172.21.15.117:0/2309849018 >> 172.21.15.117:0/2309849018 conn(0x7f498006eee0 msgr2=0x7f498006f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:33.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.972+0000 7f4985cb4640 1 -- 172.21.15.117:0/2309849018 shutdown_connections 2024-09-17T05:36:33.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.972+0000 7f4985cb4640 1 -- 172.21.15.117:0/2309849018 wait complete. 2024-09-17T05:36:33.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.974+0000 7f4985cb4640 1 Processor -- start 2024-09-17T05:36:33.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.975+0000 7f4985cb4640 1 -- start start 2024-09-17T05:36:33.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.976+0000 7f4985cb4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49800743c0 0x7f4980152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.977+0000 7f4985cb4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49800c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.977+0000 7f4985cb4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f49800c7890 0x7f498013d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.977+0000 7f497f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49800743c0 0x7f4980152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.977+0000 7f497f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49800743c0 0x7f4980152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44846/0 (socket says 172.21.15.117:44846) 2024-09-17T05:36:33.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.977+0000 7f497ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f49800c7890 0x7f498013d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f4985cb4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f49800c9610 con 0x7f49800c1eb0 2024-09-17T05:36:33.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f4985cb4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f49800b3200 con 0x7f49800743c0 2024-09-17T05:36:33.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f4985cb4640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4980157630 con 0x7f49800c7890 2024-09-17T05:36:33.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f497effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49800c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:33.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f497effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49800c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34818/0 (socket says 172.21.15.117:34818) 2024-09-17T05:36:33.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f497effd640 1 -- 172.21.15.117:0/2530882175 learned_addr learned my addr 172.21.15.117:0/2530882175 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:33.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.978+0000 7f497effd640 1 -- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f49800c7890 msgr2=0x7f498013d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.979+0000 7f497effd640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f49800c7890 0x7f498013d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.979+0000 7f497effd640 1 -- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49800743c0 msgr2=0x7f4980152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:33.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.979+0000 7f497effd640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49800743c0 0x7f4980152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:33.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.979+0000 7f497effd640 1 -- 172.21.15.117:0/2530882175 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4980150050 con 0x7f49800c1eb0 2024-09-17T05:36:33.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.979+0000 7f497effd640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49800c60a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f497000b410 tx=0x7f497000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:33.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.980+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970015020 con 0x7f49800c1eb0 2024-09-17T05:36:33.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.980+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4970011940 con 0x7f49800c1eb0 2024-09-17T05:36:33.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.980+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970011d90 con 0x7f49800c1eb0 2024-09-17T05:36:33.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.980+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498014fd90 con 0x7f49800c1eb0 2024-09-17T05:36:33.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.980+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4980151f20 con 0x7f49800c1eb0 2024-09-17T05:36:33.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.986+0000 7f497f7fe640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49800743c0 0x7f4980152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-17T05:36:33.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.987+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4960001030 con 0x7f49800c1eb0 2024-09-17T05:36:33.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.989+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4970018020 con 0x7f49800c1eb0 2024-09-17T05:36:33.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.990+0000 7f4967fff640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f496c078c10 0x7f496c07b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.990+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f497005e620 con 0x7f49800c1eb0 2024-09-17T05:36:33.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.990+0000 7f4967fff640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f496c083410 0x7f496c085900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:33.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.990+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 --> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f496c085fd0 con 0x7f496c083410 2024-09-17T05:36:33.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.990+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f497005eba0 con 0x7f49800c1eb0 2024-09-17T05:36:33.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:33.996+0000 7f497ffff640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f496c083410 0x7f496c085900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:34.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.001+0000 7f497f7fe640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f496c078c10 0x7f496c07b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:34.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.007+0000 7f497ffff640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f496c083410 0x7f496c085900 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:34.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.009+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== osd.1 v2:172.21.15.117:6808/2042501374 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f496c085fd0 con 0x7f496c083410 2024-09-17T05:36:34.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.009+0000 7f497f7fe640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f496c078c10 0x7f496c07b120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4978036000 tx=0x7f4978004ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:34.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.048+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 --> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4960002e30 con 0x7f496c083410 2024-09-17T05:36:34.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.050+0000 7f4967fff640 1 -- 172.21.15.117:0/2530882175 <== osd.1 v2:172.21.15.117:6808/2042501374 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4960002e30 con 0x7f496c083410 2024-09-17T05:36:34.056 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:34.056 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:34.056 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.055+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f496c083410 msgr2=0x7f496c085900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.055+0000 7f4985cb4640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:6808/2042501374,v1:172.21.15.117:6809/2042501374] conn(0x7f496c083410 0x7f496c085900 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.055+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f496c078c10 msgr2=0x7f496c07b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.055+0000 7f4985cb4640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f496c078c10 0x7f496c07b120 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4978036000 tx=0x7f4978004ab0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.055+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 msgr2=0x7f49800c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.056+0000 7f4985cb4640 1 --2- 172.21.15.117:0/2530882175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49800c1eb0 0x7f49800c60a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f497000b410 tx=0x7f497000b8d0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.057+0000 7f497ffff640 1 -- 172.21.15.117:0/2530882175 reap_dead start 2024-09-17T05:36:34.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.057+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 shutdown_connections 2024-09-17T05:36:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.057+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 >> 172.21.15.117:0/2530882175 conn(0x7f498006eee0 msgr2=0x7f4980068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:34.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.059+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 shutdown_connections 2024-09-17T05:36:34.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.066+0000 7f4985cb4640 1 -- 172.21.15.117:0/2530882175 wait complete. 2024-09-17T05:36:34.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 -- 172.21.15.117:0/1279135113 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc151cc0 msgr2=0x7f93cc06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 --2- 172.21.15.117:0/1279135113 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc151cc0 0x7f93cc06cda0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f93cc072610 tx=0x7f93c002a4f0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 -- 172.21.15.117:0/1279135113 shutdown_connections 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 --2- 172.21.15.117:0/1279135113 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc151cc0 0x7f93cc06cda0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f93cc072610 tx=0x7f93c002a4f0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 --2- 172.21.15.117:0/1279135113 >> v2:172.21.15.119:3300/0 conn(0x7f93cc0c88c0 0x7f93cc15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 -- 172.21.15.117:0/1279135113 >> 172.21.15.117:0/1279135113 conn(0x7f93cc0cc460 msgr2=0x7f93cc0cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 -- 172.21.15.117:0/1279135113 shutdown_connections 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.318+0000 7f93d1382640 1 -- 172.21.15.117:0/1279135113 wait complete. 2024-09-17T05:36:34.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.321+0000 7f93d1382640 1 Processor -- start 2024-09-17T05:36:34.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.324+0000 7f93d1382640 1 -- start start 2024-09-17T05:36:34.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93d1382640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 0x7f93cc1e4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93d1382640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc0c88c0 0x7f93cc150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93cbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 0x7f93cc1e4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:34.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93cbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 0x7f93cc1e4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34820/0 (socket says 172.21.15.117:34820) 2024-09-17T05:36:34.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93d1382640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc1465a0 0x7f93cc1fef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93d1382640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc15eb60 con 0x7f93cc0c6e10 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93d1382640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc15e880 con 0x7f93cc1465a0 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93d1382640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f93cc073a20 con 0x7f93cc0c88c0 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.327+0000 7f93cbfff640 1 -- 172.21.15.117:0/3792620870 learned_addr learned my addr 172.21.15.117:0/3792620870 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.328+0000 7f93cbfff640 1 -- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc0c88c0 msgr2=0x7f93cc150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.328+0000 7f93cbfff640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc0c88c0 0x7f93cc150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.328+0000 7f93cbfff640 1 -- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc1465a0 msgr2=0x7f93cc1fef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.328+0000 7f93cbfff640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc1465a0 0x7f93cc1fef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.328+0000 7f93cbfff640 1 -- 172.21.15.117:0/3792620870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc0c6480 con 0x7f93cc0c6e10 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.328+0000 7f93cbfff640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 0x7f93cc1e4f10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00ea60 tx=0x7f93bc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.329+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00cc30 con 0x7f93cc0c6e10 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.329+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93cc2021e0 con 0x7f93cc0c6e10 2024-09-17T05:36:34.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.330+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93bc00cdf0 con 0x7f93cc0c6e10 2024-09-17T05:36:34.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.330+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc005280 con 0x7f93cc0c6e10 2024-09-17T05:36:34.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.330+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93cc1591e0 con 0x7f93cc0c6e10 2024-09-17T05:36:34.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.332+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f93bc005420 con 0x7f93cc0c6e10 2024-09-17T05:36:34.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.332+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93cc073800 con 0x7f93cc0c6e10 2024-09-17T05:36:34.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.333+0000 7f93c8ff9640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b8078c90 0x7f93b807b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.333+0000 7f93cb7fe640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b8078c90 0x7f93b807b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:34.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.334+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f93bc09bbe0 con 0x7f93cc0c6e10 2024-09-17T05:36:34.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.334+0000 7f93cb7fe640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b8078c90 0x7f93b807b1a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f93c0034000 tx=0x7f93c00046c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:34.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.344+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f93bc063b30 con 0x7f93cc0c6e10 2024-09-17T05:36:34.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.715+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f93cc06dd40 con 0x7f93b8078c90 2024-09-17T05:36:34.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.718+0000 7f93c8ff9640 1 -- 172.21.15.117:0/3792620870 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f93cc06dd40 con 0x7f93b8078c90 2024-09-17T05:36:34.720 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T05:36:34.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b8078c90 msgr2=0x7f93b807b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b8078c90 0x7f93b807b1a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f93c0034000 tx=0x7f93c00046c0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 msgr2=0x7f93cc1e4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 0x7f93cc1e4f10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00ea60 tx=0x7f93bc00ef20 comp rx=0 tx=0).stop 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 shutdown_connections 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f93b8078c90 0x7f93b807b1a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc1465a0 0x7f93cc1fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93cc0c88c0 0x7f93cc150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 --2- 172.21.15.117:0/3792620870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93cc0c6e10 0x7f93cc1e4f10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 >> 172.21.15.117:0/3792620870 conn(0x7f93cc0cc460 msgr2=0x7f93cc14e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 shutdown_connections 2024-09-17T05:36:34.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.723+0000 7f93d1382640 1 -- 172.21.15.117:0/3792620870 wait complete. 2024-09-17T05:36:34.743 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T05:36:34.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:34.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/1843600602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa340c78c0 msgr2=0x7ffa200052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3bfc3640 1 --2- 172.21.15.117:0/1843600602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa340c78c0 0x7ffa200052b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffa340733b0 tx=0x7ffa20005830 comp rx=0 tx=0).stop 2024-09-17T05:36:34.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3a560640 1 -- 172.21.15.117:0/1843600602 reap_dead start 2024-09-17T05:36:34.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/1843600602 shutdown_connections 2024-09-17T05:36:34.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/1843600602 >> 172.21.15.117:0/1843600602 conn(0x7ffa3406d190 msgr2=0x7ffa3406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:34.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/1843600602 shutdown_connections 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.983+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/1843600602 wait complete. 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.986+0000 7ffa3bfc3640 1 Processor -- start 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.987+0000 7ffa3bfc3640 1 -- start start 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3bfc3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa340c78c0 0x7ffa34152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3bfc3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa34153960 0x7ffa34151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3bfc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa34158430 0x7ffa3415d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3bfc3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3415eae0 con 0x7ffa34158430 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3bfc3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffa340c86d0 con 0x7ffa340c78c0 2024-09-17T05:36:34.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3bfc3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ffa340735f0 con 0x7ffa34153960 2024-09-17T05:36:34.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3a560640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa34158430 0x7ffa3415d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:34.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3a560640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa34158430 0x7ffa3415d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34834/0 (socket says 172.21.15.117:34834) 2024-09-17T05:36:34.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.989+0000 7ffa3a560640 1 -- 172.21.15.117:0/3353314351 learned_addr learned my addr 172.21.15.117:0/3353314351 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.990+0000 7ffa3a560640 1 -- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa34153960 msgr2=0x7ffa34151f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.990+0000 7ffa3a560640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa34153960 0x7ffa34151f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.990+0000 7ffa3a560640 1 -- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa340c78c0 msgr2=0x7ffa34152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.990+0000 7ffa3a560640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa340c78c0 0x7ffa34152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.990+0000 7ffa3a560640 1 -- 172.21.15.117:0/3353314351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3414d1f0 con 0x7ffa34158430 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.990+0000 7ffa3a560640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa34158430 0x7ffa3415d4b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c650 tx=0x7ffa2c004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.991+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c02bb80 con 0x7ffa34158430 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.991+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa2c02bd20 con 0x7ffa34158430 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.991+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/3353314351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3414d4b0 con 0x7ffa34158430 2024-09-17T05:36:34.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.991+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/3353314351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa34155cc0 con 0x7ffa34158430 2024-09-17T05:36:34.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.991+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0062d0 con 0x7ffa34158430 2024-09-17T05:36:34.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:34.993+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/3353314351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa34074ff0 con 0x7ffa34158430 2024-09-17T05:36:35.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.000+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ffa2c03c030 con 0x7ffa34158430 2024-09-17T05:36:35.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.001+0000 7ffa2a7fc640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ffa18078b90 0x7ffa1807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.001+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7ffa2c0bb660 con 0x7ffa34158430 2024-09-17T05:36:35.002 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.002+0000 7ffa2a7fc640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7ffa18082700 0x7ffa18084bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.002+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 --> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa180852a0 con 0x7ffa18082700 2024-09-17T05:36:35.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.002+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ffa2c0bba50 con 0x7ffa34158430 2024-09-17T05:36:35.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.004+0000 7ffa39d5f640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ffa18078b90 0x7ffa1807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.007 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.007+0000 7ffa3955e640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7ffa18082700 0x7ffa18084bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.007+0000 7ffa3955e640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7ffa18082700 0x7ffa18084bd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:35.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.009+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== osd.2 v2:172.21.15.117:6800/2702715806 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffa180852a0 con 0x7ffa18082700 2024-09-17T05:36:35.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.009+0000 7ffa39d5f640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ffa18078b90 0x7ffa1807b0a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009ad0 tx=0x7ffa24009590 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:35.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.033+0000 7ffa3bfc3640 1 -- 172.21.15.117:0/3353314351 --> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffa34153190 con 0x7ffa18082700 2024-09-17T05:36:35.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.033+0000 7ffa2a7fc640 1 -- 172.21.15.117:0/3353314351 <== osd.2 v2:172.21.15.117:6800/2702715806 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffa34153190 con 0x7ffa18082700 2024-09-17T05:36:35.034 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:35.034 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:35.034 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:35.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.034+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7ffa18082700 msgr2=0x7ffa18084bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.035+0000 7ffa17fff640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:6800/2702715806,v1:172.21.15.117:6801/2702715806] conn(0x7ffa18082700 0x7ffa18084bd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.035+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ffa18078b90 msgr2=0x7ffa1807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.035+0000 7ffa17fff640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ffa18078b90 0x7ffa1807b0a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009ad0 tx=0x7ffa24009590 comp rx=0 tx=0).stop 2024-09-17T05:36:35.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.035+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa34158430 msgr2=0x7ffa3415d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.035+0000 7ffa17fff640 1 --2- 172.21.15.117:0/3353314351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa34158430 0x7ffa3415d4b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00c650 tx=0x7ffa2c004bb0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.035+0000 7ffa3a560640 1 -- 172.21.15.117:0/3353314351 reap_dead start 2024-09-17T05:36:35.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.038+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 shutdown_connections 2024-09-17T05:36:35.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.038+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 >> 172.21.15.117:0/3353314351 conn(0x7ffa3406d190 msgr2=0x7ffa34070fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:35.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.043+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 shutdown_connections 2024-09-17T05:36:35.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.043+0000 7ffa17fff640 1 -- 172.21.15.117:0/3353314351 wait complete. 2024-09-17T05:36:35.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T05:36:35.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.310+0000 7ff61e60a640 1 -- 172.21.15.117:0/1192044778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180b4870 msgr2=0x7ff6180b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.310+0000 7ff61e60a640 1 --2- 172.21.15.117:0/1192044778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180b4870 0x7ff6180b4c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009830 tx=0x7ff60c02a530 comp rx=0 tx=0).stop 2024-09-17T05:36:35.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.314+0000 7ff61e60a640 1 -- 172.21.15.117:0/1192044778 shutdown_connections 2024-09-17T05:36:35.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.314+0000 7ff61e60a640 1 --2- 172.21.15.117:0/1192044778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180b4870 0x7ff6180b4c80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.314+0000 7ff61e60a640 1 -- 172.21.15.117:0/1192044778 >> 172.21.15.117:0/1192044778 conn(0x7ff61806e170 msgr2=0x7ff61806e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:35.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.315+0000 7ff61e60a640 1 -- 172.21.15.117:0/1192044778 shutdown_connections 2024-09-17T05:36:35.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.315+0000 7ff61e60a640 1 -- 172.21.15.117:0/1192044778 wait complete. 2024-09-17T05:36:35.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.318+0000 7ff61e60a640 1 Processor -- start 2024-09-17T05:36:35.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.319+0000 7ff61e60a640 1 -- start start 2024-09-17T05:36:35.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff61e60a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6180b4870 0x7ff618202b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff61e60a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 0x7ff6180ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff617fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6180b4870 0x7ff618202b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff617fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6180b4870 0x7ff618202b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44908/0 (socket says 172.21.15.117:44908) 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff6177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 0x7ff6180ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff6177fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 0x7ff6180ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34850/0 (socket says 172.21.15.117:34850) 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.321+0000 7ff617fff640 1 -- 172.21.15.117:0/611562312 learned_addr learned my addr 172.21.15.117:0/611562312 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.322+0000 7ff61e60a640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6180cd290 0x7ff618154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.322+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180b43e0 con 0x7ff6180cbbd0 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.322+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff61815f090 con 0x7ff6180b4870 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.322+0000 7ff61cba7640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6180cd290 0x7ff618154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.322+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7ff6180cc770 con 0x7ff6180cd290 2024-09-17T05:36:35.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.326+0000 7ff6177fe640 1 -- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6180cd290 msgr2=0x7ff618154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.326+0000 7ff6177fe640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6180cd290 0x7ff618154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.326+0000 7ff6177fe640 1 -- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6180b4870 msgr2=0x7ff618202b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.326+0000 7ff6177fe640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6180b4870 0x7ff618202b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.326+0000 7ff6177fe640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff618203050 con 0x7ff6180cbbd0 2024-09-17T05:36:35.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.327+0000 7ff6177fe640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 0x7ff6180ca030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff60c031000 tx=0x7ff60c033f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:35.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.328+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c031d60 con 0x7ff6180cbbd0 2024-09-17T05:36:35.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.328+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6181585c0 con 0x7ff6180cbbd0 2024-09-17T05:36:35.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.328+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6181588a0 con 0x7ff6180cbbd0 2024-09-17T05:36:35.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.328+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff60c005020 con 0x7ff6180cbbd0 2024-09-17T05:36:35.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.331+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c005490 con 0x7ff6180cbbd0 2024-09-17T05:36:35.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.332+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff60c0047a0 con 0x7ff6180cbbd0 2024-09-17T05:36:35.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.333+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f8005290 con 0x7ff6180cbbd0 2024-09-17T05:36:35.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.343+0000 7ff614ff9640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff604078c90 0x7ff60407b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.343+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7ff60c0bc0e0 con 0x7ff6180cbbd0 2024-09-17T05:36:35.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.343+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff60c03f030 con 0x7ff6180cbbd0 2024-09-17T05:36:35.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.345+0000 7ff617fff640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff604078c90 0x7ff60407b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.349+0000 7ff617fff640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff604078c90 0x7ff60407b1a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff610001d30 tx=0x7ff610001cc0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:35.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.710+0000 7ff61e60a640 1 -- 172.21.15.117:0/611562312 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ff5f8000bc0 con 0x7ff604078c90 2024-09-17T05:36:35.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.712+0000 7ff614ff9640 1 -- 172.21.15.117:0/611562312 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff5f8000bc0 con 0x7ff604078c90 2024-09-17T05:36:35.712 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T05:36:35.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 -- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff604078c90 msgr2=0x7ff60407b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff604078c90 0x7ff60407b1a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff610001d30 tx=0x7ff610001cc0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 -- 172.21.15.117:0/611562312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 msgr2=0x7ff6180ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 0x7ff6180ca030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff60c031000 tx=0x7ff60c033f10 comp rx=0 tx=0).stop 2024-09-17T05:36:35.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 -- 172.21.15.117:0/611562312 shutdown_connections 2024-09-17T05:36:35.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7ff604078c90 0x7ff60407b1a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6180cd290 0x7ff618154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6180cbbd0 0x7ff6180ca030 secure :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff60c031000 tx=0x7ff60c033f10 comp rx=0 tx=0).stop 2024-09-17T05:36:35.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 --2- 172.21.15.117:0/611562312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6180b4870 0x7ff618202b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.716+0000 7ff5fe7fc640 1 -- 172.21.15.117:0/611562312 >> 172.21.15.117:0/611562312 conn(0x7ff61806e170 msgr2=0x7ff61815a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:35.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.717+0000 7ff5fe7fc640 1 -- 172.21.15.117:0/611562312 shutdown_connections 2024-09-17T05:36:35.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.717+0000 7ff5fe7fc640 1 -- 172.21.15.117:0/611562312 wait complete. 2024-09-17T05:36:35.740 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T05:36:35.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:35.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.967+0000 7f4a9f7fe640 1 -- 172.21.15.117:0/1188456032 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90042850 con 0x7f4aa01601b0 2024-09-17T05:36:35.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.967+0000 7f4aa71a1640 1 -- 172.21.15.117:0/1188456032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 msgr2=0x7f4aa00c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.967+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/1188456032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 0x7f4aa00c94e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009830 tx=0x7f4a9003b4c0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.968+0000 7f4aa71a1640 1 -- 172.21.15.117:0/1188456032 shutdown_connections 2024-09-17T05:36:35.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.968+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/1188456032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 0x7f4aa00c94e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.968+0000 7f4aa71a1640 1 -- 172.21.15.117:0/1188456032 >> 172.21.15.117:0/1188456032 conn(0x7f4aa006d190 msgr2=0x7f4aa006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:35.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.972+0000 7f4aa71a1640 1 -- 172.21.15.117:0/1188456032 shutdown_connections 2024-09-17T05:36:35.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.972+0000 7f4aa71a1640 1 -- 172.21.15.117:0/1188456032 wait complete. 2024-09-17T05:36:35.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.974+0000 7f4aa71a1640 1 Processor -- start 2024-09-17T05:36:35.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.975+0000 7f4aa71a1640 1 -- start start 2024-09-17T05:36:35.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa71a1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4aa00c7930 0x7f4aa01f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa4f3d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4aa00c7930 0x7f4aa01f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa4f3d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4aa00c7930 0x7f4aa01f1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34854/0 (socket says 172.21.15.117:34854) 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa4f3d640 1 -- 172.21.15.117:0/2520164030 learned_addr learned my addr 172.21.15.117:0/2520164030 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4aa0159330 0x7f4aa01528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 0x7f4aa014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa0162cd0 con 0x7f4aa00c7930 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa00c8430 con 0x7f4aa01601b0 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.977+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa00734d0 con 0x7f4aa0159330 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa573e640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 0x7f4aa014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4a9ffff640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4aa0159330 0x7f4aa01528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa4f3d640 1 -- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4aa0159330 msgr2=0x7f4aa01528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa4f3d640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4aa0159330 0x7f4aa01528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa4f3d640 1 -- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 msgr2=0x7f4aa014cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa4f3d640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aa01601b0 0x7f4aa014cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa4f3d640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa006db90 con 0x7f4aa00c7930 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.978+0000 7f4aa4f3d640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4aa00c7930 0x7f4aa01f1130 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4a9003bab0 tx=0x7f4a90003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.979+0000 7f4a9ffff640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4aa0159330 0x7f4aa01528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.979+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90005220 con 0x7f4aa00c7930 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.979+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa01f1670 con 0x7f4aa00c7930 2024-09-17T05:36:35.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.979+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa00c0830 con 0x7f4aa00c7930 2024-09-17T05:36:35.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.979+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a900054b0 con 0x7f4aa00c7930 2024-09-17T05:36:35.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.979+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90004a00 con 0x7f4aa00c7930 2024-09-17T05:36:35.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.983+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4a90004ba0 con 0x7f4aa00c7930 2024-09-17T05:36:35.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.983+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a80001030 con 0x7f4aa00c7930 2024-09-17T05:36:35.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.989+0000 7f4a9d7fa640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a8c078c90 0x7f4a8c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.989+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f4a900cd1c0 con 0x7f4aa00c7930 2024-09-17T05:36:35.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.989+0000 7f4a9d7fa640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7f4a8c082800 0x7f4a8c084cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:35.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.990+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a8c0853a0 con 0x7f4a8c082800 2024-09-17T05:36:35.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.990+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4a900d4050 con 0x7f4aa00c7930 2024-09-17T05:36:35.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.992+0000 7f4a9ffff640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a8c078c90 0x7f4a8c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.995+0000 7f4aa573e640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7f4a8c082800 0x7f4a8c084cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:35.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.997+0000 7f4a9ffff640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a8c078c90 0x7f4a8c07b1a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00cace0 tx=0x7f4a980073a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:35.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:35.997+0000 7f4aa573e640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7f4a8c082800 0x7f4a8c084cd0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:36.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.001+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== osd.3 v2:172.21.15.117:6824/1231670324 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a8c0853a0 con 0x7f4a8c082800 2024-09-17T05:36:36.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.029+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 --> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4a80002e30 con 0x7f4a8c082800 2024-09-17T05:36:36.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.031+0000 7f4a9d7fa640 1 -- 172.21.15.117:0/2520164030 <== osd.3 v2:172.21.15.117:6824/1231670324 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4a80002e30 con 0x7f4a8c082800 2024-09-17T05:36:36.032 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:36.032 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:36.032 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:36.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7f4a8c082800 msgr2=0x7f4a8c084cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:6824/1231670324,v1:172.21.15.117:6825/1231670324] conn(0x7f4a8c082800 0x7f4a8c084cd0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a8c078c90 msgr2=0x7f4a8c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f4a8c078c90 0x7f4a8c07b1a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00cace0 tx=0x7f4a980073a0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4aa00c7930 msgr2=0x7f4aa01f1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 --2- 172.21.15.117:0/2520164030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4aa00c7930 0x7f4aa01f1130 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4a9003bab0 tx=0x7f4a90003cd0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa573e640 1 -- 172.21.15.117:0/2520164030 reap_dead start 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 shutdown_connections 2024-09-17T05:36:36.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.032+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 >> 172.21.15.117:0/2520164030 conn(0x7f4aa006d190 msgr2=0x7f4aa0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:36.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.034+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 shutdown_connections 2024-09-17T05:36:36.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.034+0000 7f4aa71a1640 1 -- 172.21.15.117:0/2520164030 wait complete. 2024-09-17T05:36:36.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T05:36:36.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.269+0000 7f30299a1640 1 -- 172.21.15.117:0/3437992591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151760 msgr2=0x7f302406d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.270+0000 7f30299a1640 1 --2- 172.21.15.117:0/3437992591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151760 0x7f302406d2e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30240ca110 tx=0x7f301c02b5f0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.275+0000 7f30299a1640 1 -- 172.21.15.117:0/3437992591 shutdown_connections 2024-09-17T05:36:36.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.275+0000 7f30299a1640 1 --2- 172.21.15.117:0/3437992591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151760 0x7f302406d2e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.275+0000 7f30299a1640 1 --2- 172.21.15.117:0/3437992591 >> v2:172.21.15.119:3300/0 conn(0x7f3024151350 0x7f302406cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.275+0000 7f30299a1640 1 -- 172.21.15.117:0/3437992591 >> 172.21.15.117:0/3437992591 conn(0x7f30240ccbe0 msgr2=0x7f30240c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:36.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.277+0000 7f30299a1640 1 -- 172.21.15.117:0/3437992591 shutdown_connections 2024-09-17T05:36:36.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.277+0000 7f30299a1640 1 -- 172.21.15.117:0/3437992591 wait complete. 2024-09-17T05:36:36.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.282+0000 7f30299a1640 1 Processor -- start 2024-09-17T05:36:36.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.282+0000 7f30299a1640 1 -- start start 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30299a1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30240732b0 0x7f30241ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30299a1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 0x7f3024149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30299a1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3024151760 0x7f3024146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30299a1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f30240c37b0 con 0x7f3024151350 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30299a1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f30240c3630 con 0x7f30240732b0 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30299a1640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f302415a700 con 0x7f3024151760 2024-09-17T05:36:36.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30227fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 0x7f3024149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30227fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 0x7f3024149c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34884/0 (socket says 172.21.15.117:34884) 2024-09-17T05:36:36.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.284+0000 7f30227fc640 1 -- 172.21.15.117:0/1869682750 learned_addr learned my addr 172.21.15.117:0/1869682750 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:36.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.285+0000 7f30227fc640 1 -- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3024151760 msgr2=0x7f3024146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.285+0000 7f30227fc640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3024151760 0x7f3024146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.285+0000 7f30227fc640 1 -- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30240732b0 msgr2=0x7f30241ed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.285+0000 7f30227fc640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30240732b0 0x7f30241ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.285+0000 7f30227fc640 1 -- 172.21.15.117:0/1869682750 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30241edee0 con 0x7f3024151350 2024-09-17T05:36:36.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.286+0000 7f30227fc640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 0x7f3024149c10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f301800d700 tx=0x7f301800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:36.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.287+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018017070 con 0x7f3024151350 2024-09-17T05:36:36.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.287+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3018009b10 con 0x7f3024151350 2024-09-17T05:36:36.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.287+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018010460 con 0x7f3024151350 2024-09-17T05:36:36.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.289+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302414ae00 con 0x7f3024151350 2024-09-17T05:36:36.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.289+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30241dfcb0 con 0x7f3024151350 2024-09-17T05:36:36.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.292+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f302414fd00 con 0x7f3024151350 2024-09-17T05:36:36.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.303+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f301801a070 con 0x7f3024151350 2024-09-17T05:36:36.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.304+0000 7f300bfff640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f301007ae20 0x7f301007d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.304+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f301809b740 con 0x7f3024151350 2024-09-17T05:36:36.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.305+0000 7f3022ffd640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f301007ae20 0x7f301007d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.306+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f30180637a0 con 0x7f3024151350 2024-09-17T05:36:36.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.313+0000 7f3022ffd640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f301007ae20 0x7f301007d330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3014000f60 tx=0x7f30140079b0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:36.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.674+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f3024069830 con 0x7f301007ae20 2024-09-17T05:36:36.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.679+0000 7f300bfff640 1 -- 172.21.15.117:0/1869682750 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3024069830 con 0x7f301007ae20 2024-09-17T05:36:36.683 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T05:36:36.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f301007ae20 msgr2=0x7f301007d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f301007ae20 0x7f301007d330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3014000f60 tx=0x7f30140079b0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 msgr2=0x7f3024149c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 0x7f3024149c10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f301800d700 tx=0x7f301800dbc0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 shutdown_connections 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f301007ae20 0x7f301007d330 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3024151760 0x7f3024146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3024151350 0x7f3024149c10 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f301800d700 tx=0x7f301800dbc0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 --2- 172.21.15.117:0/1869682750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30240732b0 0x7f30241ed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.683+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 >> 172.21.15.117:0/1869682750 conn(0x7f30240ccbe0 msgr2=0x7f3024150320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:36.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.687+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 shutdown_connections 2024-09-17T05:36:36.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.688+0000 7f30299a1640 1 -- 172.21.15.117:0/1869682750 wait complete. 2024-09-17T05:36:36.713 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T05:36:36.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:36.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.944+0000 7f5ac27c5640 1 -- 172.21.15.117:0/1227039783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab40469d0 msgr2=0x7f5ab40db6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.944+0000 7f5ac27c5640 1 --2- 172.21.15.117:0/1227039783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab40469d0 0x7f5ab40db6c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009820 tx=0x7f5ab802a630 comp rx=0 tx=0).stop 2024-09-17T05:36:36.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.945+0000 7f5ac27c5640 1 -- 172.21.15.117:0/1227039783 shutdown_connections 2024-09-17T05:36:36.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.945+0000 7f5ac27c5640 1 --2- 172.21.15.117:0/1227039783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab40469d0 0x7f5ab40db6c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.945+0000 7f5ac27c5640 1 -- 172.21.15.117:0/1227039783 >> 172.21.15.117:0/1227039783 conn(0x7f5ab404fe30 msgr2=0x7f5ab4050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:36.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.945+0000 7f5ac27c5640 1 -- 172.21.15.117:0/1227039783 shutdown_connections 2024-09-17T05:36:36.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.945+0000 7f5ac27c5640 1 -- 172.21.15.117:0/1227039783 wait complete. 2024-09-17T05:36:36.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.948+0000 7f5ac27c5640 1 Processor -- start 2024-09-17T05:36:36.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.955+0000 7f5ac27c5640 1 -- start start 2024-09-17T05:36:36.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.958+0000 7f5ac27c5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ab400de50 0x7f5ab402bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.958+0000 7f5ac27c5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab4044cb0 0x7f5ab40d3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.958+0000 7f5ac27c5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ab40469d0 0x7f5ab40d2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.958+0000 7f5ac27c5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab40d5200 con 0x7f5ab4044cb0 2024-09-17T05:36:36.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.958+0000 7f5ac27c5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab40461d0 con 0x7f5ab40469d0 2024-09-17T05:36:36.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.958+0000 7f5ac27c5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab404bb10 con 0x7f5ab400de50 2024-09-17T05:36:36.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.961+0000 7f5ac0fc2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab4044cb0 0x7f5ab40d3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.961+0000 7f5ac0fc2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab4044cb0 0x7f5ab40d3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34904/0 (socket says 172.21.15.117:34904) 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.961+0000 7f5ac17c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ab400de50 0x7f5ab402bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.961+0000 7f5ac0fc2640 1 -- 172.21.15.117:0/147377540 learned_addr learned my addr 172.21.15.117:0/147377540 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.961+0000 7f5ac1fc4640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ab40469d0 0x7f5ab40d2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.962+0000 7f5ac0fc2640 1 -- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ab400de50 msgr2=0x7f5ab402bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.962+0000 7f5ac0fc2640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5ab400de50 0x7f5ab402bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.962+0000 7f5ac0fc2640 1 -- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ab40469d0 msgr2=0x7f5ab40d2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:36.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.962+0000 7f5ac0fc2640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ab40469d0 0x7f5ab40d2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:36.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.962+0000 7f5ac0fc2640 1 -- 172.21.15.117:0/147377540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab40d42b0 con 0x7f5ab4044cb0 2024-09-17T05:36:36.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.962+0000 7f5ac0fc2640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab4044cb0 0x7f5ab40d3d70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5abc06bdf0 tx=0x7f5abc064940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:36.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.963+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0753d0 con 0x7f5ab4044cb0 2024-09-17T05:36:36.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.963+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab40d1160 con 0x7f5ab4044cb0 2024-09-17T05:36:36.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.963+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab40d1a80 con 0x7f5ab4044cb0 2024-09-17T05:36:36.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.964+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5abc073720 con 0x7f5ab4044cb0 2024-09-17T05:36:36.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.964+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc073bb0 con 0x7f5ab4044cb0 2024-09-17T05:36:36.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.964+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a9c001030 con 0x7f5ab4044cb0 2024-09-17T05:36:36.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.967+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5abc095480 con 0x7f5ab4044cb0 2024-09-17T05:36:36.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.968+0000 7f5ab1ffb640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5aa0078c90 0x7f5aa007b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.968+0000 7f5ac17c3640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5aa0078c90 0x7f5aa007b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.968+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f5abc1174e0 con 0x7f5ab4044cb0 2024-09-17T05:36:36.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.969+0000 7f5ab1ffb640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f5aa0082810 0x7f5aa0084ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:36.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.969+0000 7f5ac17c3640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5aa0078c90 0x7f5aa007b1a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5aac007fd0 tx=0x7f5aac009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:36.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.969+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 --> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5abc08fec0 con 0x7f5aa0082810 2024-09-17T05:36:36.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.969+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5abc11e050 con 0x7f5ab4044cb0 2024-09-17T05:36:36.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.971+0000 7f5ac1fc4640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f5aa0082810 0x7f5aa0084ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:36.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.977+0000 7f5ac1fc4640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f5aa0082810 0x7f5aa0084ce0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:36.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:36.982+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== osd.4 v2:172.21.15.119:6810/4200593796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5abc08fec0 con 0x7f5aa0082810 2024-09-17T05:36:37.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.024+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 --> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5a9c002e30 con 0x7f5aa0082810 2024-09-17T05:36:37.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.027+0000 7f5ab1ffb640 1 -- 172.21.15.117:0/147377540 <== osd.4 v2:172.21.15.119:6810/4200593796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5a9c002e30 con 0x7f5aa0082810 2024-09-17T05:36:37.028 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:37.028 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:37.028 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:37.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.028+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f5aa0082810 msgr2=0x7f5aa0084ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.028+0000 7f5ac27c5640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6810/4200593796,v1:172.21.15.119:6811/4200593796] conn(0x7f5aa0082810 0x7f5aa0084ce0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.028+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5aa0078c90 msgr2=0x7f5aa007b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.028+0000 7f5ac27c5640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f5aa0078c90 0x7f5aa007b1a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5aac007fd0 tx=0x7f5aac009040 comp rx=0 tx=0).stop 2024-09-17T05:36:37.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.029+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab4044cb0 msgr2=0x7f5ab40d3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.029+0000 7f5ac27c5640 1 --2- 172.21.15.117:0/147377540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ab4044cb0 0x7f5ab40d3d70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5abc06bdf0 tx=0x7f5abc064940 comp rx=0 tx=0).stop 2024-09-17T05:36:37.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.029+0000 7f5ac1fc4640 1 -- 172.21.15.117:0/147377540 reap_dead start 2024-09-17T05:36:37.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.030+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 shutdown_connections 2024-09-17T05:36:37.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.030+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 >> 172.21.15.117:0/147377540 conn(0x7f5ab404fe30 msgr2=0x7f5ab4034290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:37.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.031+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 shutdown_connections 2024-09-17T05:36:37.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.031+0000 7f5ac27c5640 1 -- 172.21.15.117:0/147377540 wait complete. 2024-09-17T05:36:37.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T05:36:37.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.275+0000 7fefa3fff640 1 -- 172.21.15.117:0/3695665258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 msgr2=0x7fef9c006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.275+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3695665258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c006010 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00d620 tx=0x7fef9802a5f0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.276+0000 7fefa3fff640 1 -- 172.21.15.117:0/3695665258 shutdown_connections 2024-09-17T05:36:37.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.276+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3695665258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c006010 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.276+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3695665258 >> v2:172.21.15.119:3300/0 conn(0x7fef9c044d20 0x7fef9c0454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.276+0000 7fefa3fff640 1 -- 172.21.15.117:0/3695665258 >> 172.21.15.117:0/3695665258 conn(0x7fef9c04fe30 msgr2=0x7fef9c050240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:37.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.277+0000 7fefa3fff640 1 -- 172.21.15.117:0/3695665258 shutdown_connections 2024-09-17T05:36:37.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.277+0000 7fefa3fff640 1 -- 172.21.15.117:0/3695665258 wait complete. 2024-09-17T05:36:37.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.281+0000 7fefa3fff640 1 Processor -- start 2024-09-17T05:36:37.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.281+0000 7fefa3fff640 1 -- start start 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c1796e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c044d20 0x7fef9c0d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa3fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c0df620 0x7fef9c0d9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa3fff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c0465d0 con 0x7fef9c005c00 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa3fff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c006550 con 0x7fef9c044d20 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa3fff640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fef9c0468b0 con 0x7fef9c0df620 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c1796e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c1796e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34926/0 (socket says 172.21.15.117:34926) 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.283+0000 7fefa2ffd640 1 -- 172.21.15.117:0/3140584595 learned_addr learned my addr 172.21.15.117:0/3140584595 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:37.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.284+0000 7fefa37fe640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c0df620 0x7fef9c0d9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.285+0000 7fefa2ffd640 1 -- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c0df620 msgr2=0x7fef9c0d9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.285+0000 7fefa2ffd640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c0df620 0x7fef9c0d9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.285+0000 7fefa2ffd640 1 -- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c044d20 msgr2=0x7fef9c0d65b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:36:37.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.285+0000 7fefa2ffd640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c044d20 0x7fef9c0d65b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.285+0000 7fefa2ffd640 1 -- 172.21.15.117:0/3140584595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9c0d5510 con 0x7fef9c005c00 2024-09-17T05:36:37.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.285+0000 7fefa2ffd640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c1796e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fef9400b410 tx=0x7fef9400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:37.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.287+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94015020 con 0x7fef9c005c00 2024-09-17T05:36:37.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.287+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef94011920 con 0x7fef9c005c00 2024-09-17T05:36:37.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.287+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94011d70 con 0x7fef9c005c00 2024-09-17T05:36:37.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.288+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9c0d57f0 con 0x7fef9c005c00 2024-09-17T05:36:37.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.289+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9c0d5db0 con 0x7fef9c005c00 2024-09-17T05:36:37.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.290+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9c0dee40 con 0x7fef9c005c00 2024-09-17T05:36:37.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.292+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef94018020 con 0x7fef9c005c00 2024-09-17T05:36:37.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.293+0000 7fef8b7fe640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef90078be0 0x7fef9007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.293+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fef9409adc0 con 0x7fef9c005c00 2024-09-17T05:36:37.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.294+0000 7fefa27fc640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef90078be0 0x7fef9007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.296+0000 7fefa27fc640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef90078be0 0x7fef9007b0f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefa4052910 tx=0x7fefa406ab80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:37.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.310+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef9c0dee40 con 0x7fef9c005c00 2024-09-17T05:36:37.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.684+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fef9c04adb0 con 0x7fef90078be0 2024-09-17T05:36:37.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.690+0000 7fef8b7fe640 1 -- 172.21.15.117:0/3140584595 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fef9c04adb0 con 0x7fef90078be0 2024-09-17T05:36:37.691 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T05:36:37.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.693+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef90078be0 msgr2=0x7fef9007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.693+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef90078be0 0x7fef9007b0f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefa4052910 tx=0x7fefa406ab80 comp rx=0 tx=0).stop 2024-09-17T05:36:37.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.693+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 msgr2=0x7fef9c1796e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.693+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c1796e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fef9400b410 tx=0x7fef9400b8d0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 shutdown_connections 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fef90078be0 0x7fef9007b0f0 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefa4052910 tx=0x7fefa406ab80 comp rx=0 tx=0).stop 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fef9c0df620 0x7fef9c0d9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef9c044d20 0x7fef9c0d65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 --2- 172.21.15.117:0/3140584595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c005c00 0x7fef9c1796e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 >> 172.21.15.117:0/3140584595 conn(0x7fef9c04fe30 msgr2=0x7fef9c0d3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 shutdown_connections 2024-09-17T05:36:37.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.694+0000 7fefa3fff640 1 -- 172.21.15.117:0/3140584595 wait complete. 2024-09-17T05:36:37.715 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T05:36:37.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:37.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.935+0000 7f8718ef5640 1 -- 172.21.15.117:0/2964091270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 msgr2=0x7f870c0db6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.935+0000 7f8718ef5640 1 --2- 172.21.15.117:0/2964091270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0db6b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f870c034250 tx=0x7f870802b600 comp rx=0 tx=0).stop 2024-09-17T05:36:37.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.937+0000 7f8718ef5640 1 -- 172.21.15.117:0/2964091270 shutdown_connections 2024-09-17T05:36:37.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.937+0000 7f8718ef5640 1 --2- 172.21.15.117:0/2964091270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0db6b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.937+0000 7f8718ef5640 1 -- 172.21.15.117:0/2964091270 >> 172.21.15.117:0/2964091270 conn(0x7f870c04fe20 msgr2=0x7f870c050230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:37.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.941+0000 7f8718ef5640 1 -- 172.21.15.117:0/2964091270 shutdown_connections 2024-09-17T05:36:37.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.942+0000 7f8718ef5640 1 -- 172.21.15.117:0/2964091270 wait complete. 2024-09-17T05:36:37.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.944+0000 7f8718ef5640 1 Processor -- start 2024-09-17T05:36:37.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.945+0000 7f8718ef5640 1 -- start start 2024-09-17T05:36:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.947+0000 7f8718ef5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f87137fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f87137fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0ee800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34936/0 (socket says 172.21.15.117:34936) 2024-09-17T05:36:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f87137fe640 1 -- 172.21.15.117:0/2183070383 learned_addr learned my addr 172.21.15.117:0/2183070383 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f8718ef5640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f870c0469c0 0x7f870c047340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f8718ef5640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f870c0d1690 0x7f870c048270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f8718ef5640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f870c045ee0 con 0x7f870c044ca0 2024-09-17T05:36:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f8718ef5640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f870c045c00 con 0x7f870c0469c0 2024-09-17T05:36:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.948+0000 7f8718ef5640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f870c0461c0 con 0x7f870c0d1690 2024-09-17T05:36:37.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.950+0000 7f8712ffd640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f870c0469c0 0x7f870c047340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.951+0000 7f8712ffd640 1 -- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f870c0d1690 msgr2=0x7f870c048270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.951+0000 7f8712ffd640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f870c0d1690 0x7f870c048270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.951+0000 7f8712ffd640 1 -- 172.21.15.117:0/2183070383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 msgr2=0x7f870c0ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:37.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.951+0000 7f8712ffd640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:37.952 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.951+0000 7f8712ffd640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c0eed40 con 0x7f870c0469c0 2024-09-17T05:36:37.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.952+0000 7f8712ffd640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f870c0469c0 0x7f870c047340 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f870000b440 tx=0x7f870000b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:37.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.952+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700011830 con 0x7f870c0469c0 2024-09-17T05:36:37.954 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.952+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8700016070 con 0x7f870c0469c0 2024-09-17T05:36:37.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.953+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700011d40 con 0x7f870c0469c0 2024-09-17T05:36:37.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.953+0000 7f87137fe640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f870c044ca0 0x7f870c0ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:36:37.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.953+0000 7f8718ef5640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870c047880 con 0x7f870c0469c0 2024-09-17T05:36:37.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.953+0000 7f8718ef5640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870c0d47f0 con 0x7f870c0469c0 2024-09-17T05:36:37.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.955+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f870c050390 con 0x7f870c0469c0 2024-09-17T05:36:37.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.957+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f870001b460 con 0x7f870c0469c0 2024-09-17T05:36:37.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.957+0000 7f86fffff640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f86f8078c60 0x7f86f807b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.960+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f870009b8a0 con 0x7f870c0469c0 2024-09-17T05:36:37.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.960+0000 7f86fffff640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7f86f80827d0 0x7f86f8084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:37.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.960+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f870001de70 con 0x7f86f80827d0 2024-09-17T05:36:37.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.960+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f870009bc90 con 0x7f870c0469c0 2024-09-17T05:36:37.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.961+0000 7f87137fe640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f86f8078c60 0x7f86f807b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.961+0000 7f8713fff640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7f86f80827d0 0x7f86f8084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:37.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.967+0000 7f8713fff640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7f86f80827d0 0x7f86f8084ca0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:37.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.971+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== osd.5 v2:172.21.15.119:6802/2281253283 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f870001de70 con 0x7f86f80827d0 2024-09-17T05:36:37.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:37.979+0000 7f87137fe640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f86f8078c60 0x7f86f807b170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8708009f50 tx=0x7f8708004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.008+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 --> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f870c050390 con 0x7f86f80827d0 2024-09-17T05:36:38.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.009+0000 7f86fffff640 1 -- 172.21.15.117:0/2183070383 <== osd.5 v2:172.21.15.119:6802/2281253283 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f870c050390 con 0x7f86f80827d0 2024-09-17T05:36:38.010 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:38.010 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:38.010 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:38.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.010+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7f86f80827d0 msgr2=0x7f86f8084ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.010+0000 7f86fdffb640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6802/2281253283,v1:172.21.15.119:6803/2281253283] conn(0x7f86f80827d0 0x7f86f8084ca0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.013+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f86f8078c60 msgr2=0x7f86f807b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.014+0000 7f86fdffb640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f86f8078c60 0x7f86f807b170 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8708009f50 tx=0x7f8708004730 comp rx=0 tx=0).stop 2024-09-17T05:36:38.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.014+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f870c0469c0 msgr2=0x7f870c047340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.014 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.014+0000 7f86fdffb640 1 --2- 172.21.15.117:0/2183070383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f870c0469c0 0x7f870c047340 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f870000b440 tx=0x7f870000b900 comp rx=0 tx=0).stop 2024-09-17T05:36:38.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.014+0000 7f8713fff640 1 -- 172.21.15.117:0/2183070383 reap_dead start 2024-09-17T05:36:38.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.015+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 shutdown_connections 2024-09-17T05:36:38.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.015+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 >> 172.21.15.117:0/2183070383 conn(0x7f870c04fe20 msgr2=0x7f870c0d8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.022+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 shutdown_connections 2024-09-17T05:36:38.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.022+0000 7f86fdffb640 1 -- 172.21.15.117:0/2183070383 wait complete. 2024-09-17T05:36:38.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 -- 172.21.15.117:0/2967696771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe008153260 msgr2=0x7fe0080c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 --2- 172.21.15.117:0/2967696771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe008153260 0x7fe0080c37c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009860 tx=0x7fdff402a4f0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 -- 172.21.15.117:0/2967696771 shutdown_connections 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 --2- 172.21.15.117:0/2967696771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe008153260 0x7fe0080c37c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 -- 172.21.15.117:0/2967696771 >> 172.21.15.117:0/2967696771 conn(0x7fe008076430 msgr2=0x7fe008072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 -- 172.21.15.117:0/2967696771 shutdown_connections 2024-09-17T05:36:38.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.278+0000 7fe00ebba640 1 -- 172.21.15.117:0/2967696771 wait complete. 2024-09-17T05:36:38.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.279+0000 7fe00ebba640 1 Processor -- start 2024-09-17T05:36:38.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.279+0000 7fe00ebba640 1 -- start start 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00ebba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0080cad30 0x7fe008153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00ebba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 0x7fe00815a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00ebba640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe00815d220 0x7fe00814e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00ebba640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe0080cbc30 con 0x7fe008153260 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00ebba640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe00815cd00 con 0x7fe0080cad30 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00ebba640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fe0080c9e40 con 0x7fe00815d220 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 0x7fe00815a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 0x7fe00815a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34944/0 (socket says 172.21.15.117:34944) 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 -- 172.21.15.117:0/3209489277 learned_addr learned my addr 172.21.15.117:0/3209489277 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00c956640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0080cad30 0x7fe008153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00d157640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe00815d220 0x7fe00814e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 -- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe00815d220 msgr2=0x7fe00814e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe00815d220 0x7fe00814e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 -- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0080cad30 msgr2=0x7fe008153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0080cad30 0x7fe008153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe007fff640 1 -- 172.21.15.117:0/3209489277 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00815ac40 con 0x7fe008153260 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00d157640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe00815d220 0x7fe00814e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.281+0000 7fe00c956640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0080cad30 0x7fe008153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.282+0000 7fe007fff640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 0x7fe00815a700 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdff402aae0 tx=0x7fdff4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.282+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4031b50 con 0x7fe008153260 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.282+0000 7fe00ebba640 1 -- 172.21.15.117:0/3209489277 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0081475f0 con 0x7fe008153260 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.282+0000 7fe00ebba640 1 -- 172.21.15.117:0/3209489277 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe008155f90 con 0x7fe008153260 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.282+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdff4031de0 con 0x7fe008153260 2024-09-17T05:36:38.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.282+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4048060 con 0x7fe008153260 2024-09-17T05:36:38.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.283+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe0005290 con 0x7fe008153260 2024-09-17T05:36:38.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.284+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdff402ad10 con 0x7fe008153260 2024-09-17T05:36:38.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.284+0000 7fe0057fa640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfec078b90 0x7fdfec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.284+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fdff40bc1e0 con 0x7fe008153260 2024-09-17T05:36:38.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.284+0000 7fe00c956640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfec078b90 0x7fdfec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.285+0000 7fe00c956640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfec078b90 0x7fdfec07b0a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdffc000f60 tx=0x7fdffc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.289+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdff403f030 con 0x7fe008153260 2024-09-17T05:36:38.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.467+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fdfe0000bc0 con 0x7fdfec078b90 2024-09-17T05:36:38.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.468+0000 7fe0057fa640 1 -- 172.21.15.117:0/3209489277 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdfe0000bc0 con 0x7fdfec078b90 2024-09-17T05:36:38.468 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfec078b90 msgr2=0x7fdfec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfec078b90 0x7fdfec07b0a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdffc000f60 tx=0x7fdffc009040 comp rx=0 tx=0).stop 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 msgr2=0x7fe00815a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 0x7fe00815a700 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdff402aae0 tx=0x7fdff4033ba0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 shutdown_connections 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fdfec078b90 0x7fdfec07b0a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe00815d220 0x7fe00814e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe008153260 0x7fe00815a700 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 --2- 172.21.15.117:0/3209489277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0080cad30 0x7fe008153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 >> 172.21.15.117:0/3209489277 conn(0x7fe008076430 msgr2=0x7fe00814f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 shutdown_connections 2024-09-17T05:36:38.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.470+0000 7fdff2ffd640 1 -- 172.21.15.117:0/3209489277 wait complete. 2024-09-17T05:36:38.482 INFO:tasks.ceph:Scrubbing osd.6 2024-09-17T05:36:38.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 -- 172.21.15.117:0/3404479463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d40c4460 msgr2=0x7fc9d415c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 --2- 172.21.15.117:0/3404479463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d40c4460 0x7fc9d415c5a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0005fd0 tx=0x7fc9d002a5d0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 -- 172.21.15.117:0/3404479463 shutdown_connections 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 --2- 172.21.15.117:0/3404479463 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d40cbdc0 0x7fc9d4159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 --2- 172.21.15.117:0/3404479463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d40c4460 0x7fc9d415c5a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 -- 172.21.15.117:0/3404479463 >> 172.21.15.117:0/3404479463 conn(0x7fc9d40703e0 msgr2=0x7fc9d40707f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 -- 172.21.15.117:0/3404479463 shutdown_connections 2024-09-17T05:36:38.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.598+0000 7fc9dbda8640 1 -- 172.21.15.117:0/3404479463 wait complete. 2024-09-17T05:36:38.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.600+0000 7fc9dbda8640 1 Processor -- start 2024-09-17T05:36:38.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.600+0000 7fc9dbda8640 1 -- start start 2024-09-17T05:36:38.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9dbda8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9d40cbdc0 0x7fc9d414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9dbda8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d4153170 0x7fc9d415ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9dbda8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d41508f0 0x7fc9d40b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9dbda8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d4159270 con 0x7fc9d41508f0 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9dbda8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d40cade0 con 0x7fc9d40cbdc0 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9dbda8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d40ca440 con 0x7fc9d4153170 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d41508f0 0x7fc9d40b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d41508f0 0x7fc9d40b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34970/0 (socket says 172.21.15.117:34970) 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 -- 172.21.15.117:0/1006172909 learned_addr learned my addr 172.21.15.117:0/1006172909 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9d9b44640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9d40cbdc0 0x7fc9d414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9d9343640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d4153170 0x7fc9d415ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 -- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d4153170 msgr2=0x7fc9d415ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc9d4153170 0x7fc9d415ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 -- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9d40cbdc0 msgr2=0x7fc9d414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9d40cbdc0 0x7fc9d414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 -- 172.21.15.117:0/1006172909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d415c920 con 0x7fc9d41508f0 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.601+0000 7fc9da345640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d41508f0 0x7fc9d40b2cb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00e990 tx=0x7fc9cc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.602+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc004d20 con 0x7fc9d41508f0 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.602+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9cc00ca40 con 0x7fc9d41508f0 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.602+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d406efb0 con 0x7fc9d41508f0 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.602+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc005020 con 0x7fc9d41508f0 2024-09-17T05:36:38.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.602+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d415c660 con 0x7fc9d41508f0 2024-09-17T05:36:38.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.602+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9d40c55b0 con 0x7fc9d41508f0 2024-09-17T05:36:38.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.603+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9cc0051c0 con 0x7fc9d41508f0 2024-09-17T05:36:38.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9ca7fc640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b8078b90 0x7fc9b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fc9cc09b820 con 0x7fc9d41508f0 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9ca7fc640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fc9b8082680 0x7fc9b8084b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 --> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9cc018ec0 con 0x7fc9b8082680 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9d9b44640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b8078b90 0x7fc9b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc9cc0a02c0 con 0x7fc9d41508f0 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.604+0000 7fc9d9343640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fc9b8082680 0x7fc9b8084b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.605+0000 7fc9d9b44640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b8078b90 0x7fc9b807b0a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0033000 tx=0x7fc9d0004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.605+0000 7fc9d9343640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fc9b8082680 0x7fc9b8084b50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.606+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== osd.6 v2:172.21.15.119:6818/2947255794 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9cc018ec0 con 0x7fc9b8082680 2024-09-17T05:36:38.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.619+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 --> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc9d4158440 con 0x7fc9b8082680 2024-09-17T05:36:38.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.619+0000 7fc9ca7fc640 1 -- 172.21.15.117:0/1006172909 <== osd.6 v2:172.21.15.119:6818/2947255794 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc9d4158440 con 0x7fc9b8082680 2024-09-17T05:36:38.620 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:38.620 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:38.621 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fc9b8082680 msgr2=0x7fc9b8084b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6818/2947255794,v1:172.21.15.119:6819/2947255794] conn(0x7fc9b8082680 0x7fc9b8084b50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b8078b90 msgr2=0x7fc9b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc9b8078b90 0x7fc9b807b0a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0033000 tx=0x7fc9d0004730 comp rx=0 tx=0).stop 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d41508f0 msgr2=0x7fc9d40b2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 --2- 172.21.15.117:0/1006172909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9d41508f0 0x7fc9d40b2cb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00e990 tx=0x7fc9cc00ee50 comp rx=0 tx=0).stop 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9da345640 1 -- 172.21.15.117:0/1006172909 reap_dead start 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 shutdown_connections 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 >> 172.21.15.117:0/1006172909 conn(0x7fc9d40703e0 msgr2=0x7fc9d4157730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 shutdown_connections 2024-09-17T05:36:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.620+0000 7fc9dbda8640 1 -- 172.21.15.117:0/1006172909 wait complete. 2024-09-17T05:36:38.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-17T05:36:38.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 -- 172.21.15.117:0/3776456332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac152210 msgr2=0x7f91ac0c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 --2- 172.21.15.117:0/3776456332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac152210 0x7f91ac0c4f50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f919c005fd0 tx=0x7f919c02a540 comp rx=0 tx=0).stop 2024-09-17T05:36:38.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 -- 172.21.15.117:0/3776456332 shutdown_connections 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 --2- 172.21.15.117:0/3776456332 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f91ac0c5490 0x7f91ac162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 --2- 172.21.15.117:0/3776456332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac152210 0x7f91ac0c4f50 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 -- 172.21.15.117:0/3776456332 >> 172.21.15.117:0/3776456332 conn(0x7f91ac06e590 msgr2=0x7f91ac0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 -- 172.21.15.117:0/3776456332 shutdown_connections 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.745+0000 7f91b32f9640 1 -- 172.21.15.117:0/3776456332 wait complete. 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.747+0000 7f91b32f9640 1 Processor -- start 2024-09-17T05:36:38.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.747+0000 7f91b32f9640 1 -- start start 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b32f9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 0x7f91ac154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b32f9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ac0c67a0 0x7f91ac0ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b32f9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f91ac152210 0x7f91ac153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b32f9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac0ce0d0 con 0x7f91ac0c5490 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b32f9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac0c58a0 con 0x7f91ac0c67a0 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b32f9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f91ac0c7ff0 con 0x7f91ac152210 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 0x7f91ac154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 0x7f91ac154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34980/0 (socket says 172.21.15.117:34980) 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 -- 172.21.15.117:0/2029792636 learned_addr learned my addr 172.21.15.117:0/2029792636 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:38.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b0894640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ac0c67a0 0x7f91ac0ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1896640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f91ac152210 0x7f91ac153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 -- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f91ac152210 msgr2=0x7f91ac153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f91ac152210 0x7f91ac153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 -- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ac0c67a0 msgr2=0x7f91ac0ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.748+0000 7f91b1095640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ac0c67a0 0x7f91ac0ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91b1095640 1 -- 172.21.15.117:0/2029792636 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ac1f2dc0 con 0x7f91ac0c5490 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91b1095640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 0x7f91ac154dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f91a800c3d0 tx=0x7f91a800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a8017070 con 0x7f91ac0c5490 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91a80053c0 con 0x7f91ac0c5490 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a8003850 con 0x7f91ac0c5490 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ac155310 con 0x7f91ac0c5490 2024-09-17T05:36:38.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.749+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ac152bc0 con 0x7f91ac0c5490 2024-09-17T05:36:38.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.750+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ac0cde20 con 0x7f91ac0c5490 2024-09-17T05:36:38.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.751+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f91a801a070 con 0x7f91ac0c5490 2024-09-17T05:36:38.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.751+0000 7f91a1ffb640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9190078b90 0x7f919007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:38.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.751+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f91a809a460 con 0x7f91ac0c5490 2024-09-17T05:36:38.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.751+0000 7f91b0894640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9190078b90 0x7f919007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:38.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.752+0000 7f91b0894640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9190078b90 0x7f919007b0a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f919c033000 tx=0x7f919c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:38.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.756+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f91a8061ea0 con 0x7f91ac0c5490 2024-09-17T05:36:38.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.933+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f91ac1f2f90 con 0x7f9190078b90 2024-09-17T05:36:38.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.934+0000 7f91a1ffb640 1 -- 172.21.15.117:0/2029792636 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f91ac1f2f90 con 0x7f9190078b90 2024-09-17T05:36:38.935 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 6 to deep-scrub 2024-09-17T05:36:38.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9190078b90 msgr2=0x7f919007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9190078b90 0x7f919007b0a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f919c033000 tx=0x7f919c004730 comp rx=0 tx=0).stop 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 msgr2=0x7f91ac154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 0x7f91ac154dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f91a800c3d0 tx=0x7f91a800c890 comp rx=0 tx=0).stop 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 shutdown_connections 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9190078b90 0x7f919007b0a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f91ac152210 0x7f91ac153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ac0c67a0 0x7f91ac0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 --2- 172.21.15.117:0/2029792636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91ac0c5490 0x7f91ac154dd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 >> 172.21.15.117:0/2029792636 conn(0x7f91ac06e590 msgr2=0x7f91ac0ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 shutdown_connections 2024-09-17T05:36:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:38.936+0000 7f91b32f9640 1 -- 172.21.15.117:0/2029792636 wait complete. 2024-09-17T05:36:38.948 INFO:tasks.ceph:Scrubbing osd.7 2024-09-17T05:36:38.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:36:39.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 -- 172.21.15.117:0/3422239508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4c4004800 msgr2=0x7fc4c4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 --2- 172.21.15.117:0/3422239508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4c4004800 0x7fc4c4004c10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8076e20 tx=0x7fc4c4005830 comp rx=0 tx=0).stop 2024-09-17T05:36:39.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 -- 172.21.15.117:0/3422239508 shutdown_connections 2024-09-17T05:36:39.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 --2- 172.21.15.117:0/3422239508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4c4004800 0x7fc4c4004c10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 -- 172.21.15.117:0/3422239508 >> 172.21.15.117:0/3422239508 conn(0x7fc4d80703e0 msgr2=0x7fc4d80707f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:39.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 -- 172.21.15.117:0/3422239508 shutdown_connections 2024-09-17T05:36:39.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.069+0000 7fc4de6be640 1 -- 172.21.15.117:0/3422239508 wait complete. 2024-09-17T05:36:39.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.071+0000 7fc4de6be640 1 Processor -- start 2024-09-17T05:36:39.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.071+0000 7fc4de6be640 1 -- start start 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4de6be640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4c4004800 0x7fc4d806f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4de6be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d815b4f0 0x7fc4d814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4de6be640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d815a170 0x7fc4d806ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4de6be640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d8159270 con 0x7fc4d815b4f0 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4de6be640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d80cade0 con 0x7fc4d815a170 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4de6be640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d80ca440 con 0x7fc4c4004800 2024-09-17T05:36:39.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4d77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d815b4f0 0x7fc4d814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4d77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d815b4f0 0x7fc4d814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:35012/0 (socket says 172.21.15.117:35012) 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.072+0000 7fc4d77fe640 1 -- 172.21.15.117:0/1201949500 learned_addr learned my addr 172.21.15.117:0/1201949500 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4dcc5b640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d815a170 0x7fc4d806ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d7fff640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4c4004800 0x7fc4d806f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d77fe640 1 -- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4c4004800 msgr2=0x7fc4d806f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d77fe640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4c4004800 0x7fc4d806f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d77fe640 1 -- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d815a170 msgr2=0x7fc4d806ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d77fe640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d815a170 0x7fc4d806ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d77fe640 1 -- 172.21.15.117:0/1201949500 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d806f120 con 0x7fc4d815b4f0 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d7fff640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4c4004800 0x7fc4d806f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:36:39.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d77fe640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d815b4f0 0x7fc4d814e830 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0001780 tx=0x7fc4c002af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0042070 con 0x7fc4d815b4f0 2024-09-17T05:36:39.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4c0039070 con 0x7fc4d815b4f0 2024-09-17T05:36:39.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d815bc70 con 0x7fc4d815b4f0 2024-09-17T05:36:39.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0003850 con 0x7fc4d815b4f0 2024-09-17T05:36:39.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.073+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d815c230 con 0x7fc4d815b4f0 2024-09-17T05:36:39.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.074+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc4d80c55b0 con 0x7fc4d815b4f0 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.075+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc4c003b070 con 0x7fc4d815b4f0 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.075+0000 7fc4d4ff9640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4b8078b90 0x7fc4b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fc4c00ba8a0 con 0x7fc4d815b4f0 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4d4ff9640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7fc4b8082680 0x7fc4b8084b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 --> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4c0043e50 con 0x7fc4b8082680 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc4c00bad30 con 0x7fc4d815b4f0 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4d7fff640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4b8078b90 0x7fc4b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4dcc5b640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7fc4b8082680 0x7fc4b8084b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4d7fff640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4b8078b90 0x7fc4b807b0a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc4d806c490 tx=0x7fc4c8006930 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.076+0000 7fc4dcc5b640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7fc4b8082680 0x7fc4b8084b50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.077+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== osd.7 v2:172.21.15.119:6826/2580827441 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc4c0043e50 con 0x7fc4b8082680 2024-09-17T05:36:39.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.089+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 --> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc4d806fe80 con 0x7fc4b8082680 2024-09-17T05:36:39.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4d4ff9640 1 -- 172.21.15.117:0/1201949500 <== osd.7 v2:172.21.15.119:6826/2580827441 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc4d806fe80 con 0x7fc4b8082680 2024-09-17T05:36:39.091 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-09-17T05:36:39.091 INFO:teuthology.orchestra.run.smithi117.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:36:39.091 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7fc4b8082680 msgr2=0x7fc4b8084b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6826/2580827441,v1:172.21.15.119:6827/2580827441] conn(0x7fc4b8082680 0x7fc4b8084b50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4b8078b90 msgr2=0x7fc4b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4b8078b90 0x7fc4b807b0a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc4d806c490 tx=0x7fc4c8006930 comp rx=0 tx=0).stop 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d815b4f0 msgr2=0x7fc4d814e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 --2- 172.21.15.117:0/1201949500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d815b4f0 0x7fc4d814e830 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0001780 tx=0x7fc4c002af40 comp rx=0 tx=0).stop 2024-09-17T05:36:39.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4dcc5b640 1 -- 172.21.15.117:0/1201949500 reap_dead start 2024-09-17T05:36:39.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 shutdown_connections 2024-09-17T05:36:39.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 >> 172.21.15.117:0/1201949500 conn(0x7fc4d80703e0 msgr2=0x7fc4d80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:39.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 shutdown_connections 2024-09-17T05:36:39.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.090+0000 7fc4de6be640 1 -- 172.21.15.117:0/1201949500 wait complete. 2024-09-17T05:36:39.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-17T05:36:39.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 -- 172.21.15.117:0/3230718494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4570071850 msgr2=0x7f4570071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 --2- 172.21.15.117:0/3230718494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4570071850 0x7f4570071c60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4564009830 tx=0x7f456402a480 comp rx=0 tx=0).stop 2024-09-17T05:36:39.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 -- 172.21.15.117:0/3230718494 shutdown_connections 2024-09-17T05:36:39.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 --2- 172.21.15.117:0/3230718494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4570071850 0x7f4570071c60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 --2- 172.21.15.117:0/3230718494 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f45700c67a0 0x7f4570152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 -- 172.21.15.117:0/3230718494 >> 172.21.15.117:0/3230718494 conn(0x7f457006e590 msgr2=0x7f45700cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:39.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 -- 172.21.15.117:0/3230718494 shutdown_connections 2024-09-17T05:36:39.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.214+0000 7f45752d5640 1 -- 172.21.15.117:0/3230718494 wait complete. 2024-09-17T05:36:39.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.216+0000 7f45752d5640 1 Processor -- start 2024-09-17T05:36:39.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.216+0000 7f45752d5640 1 -- start start 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f45752d5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4570071850 0x7f45701efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f45752d5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 0x7f457006cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f45752d5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45700c67a0 0x7f4570145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f45752d5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f45700c7ff0 con 0x7f45700c4f50 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f45752d5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f45700ce0d0 con 0x7f45700c67a0 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f45752d5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f45700c83a0 con 0x7f4570071850 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.217+0000 7f456e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 0x7f457006cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 0x7f457006cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:35038/0 (socket says 172.21.15.117:35038) 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 -- 172.21.15.117:0/2247750925 learned_addr learned my addr 172.21.15.117:0/2247750925 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456effd640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4570071850 0x7f45701efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456f7fe640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45700c67a0 0x7f4570145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 -- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4570071850 msgr2=0x7f45701efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4570071850 0x7f45701efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 -- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45700c67a0 msgr2=0x7f4570145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45700c67a0 0x7f4570145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 -- 172.21.15.117:0/2247750925 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45701f02b0 con 0x7f45700c4f50 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f456e7fc640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 0x7f457006cf80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4564009950 tx=0x7f4564033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564005c90 con 0x7f45700c4f50 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4564031e50 con 0x7f45700c4f50 2024-09-17T05:36:39.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4570145de0 con 0x7f45700c4f50 2024-09-17T05:36:39.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564033310 con 0x7f45700c4f50 2024-09-17T05:36:39.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.218+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457014e200 con 0x7f45700c4f50 2024-09-17T05:36:39.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.219+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45700c7940 con 0x7f45700c4f50 2024-09-17T05:36:39.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.220+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f45640334b0 con 0x7f45700c4f50 2024-09-17T05:36:39.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.221+0000 7f45577fe640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f455c078bc0 0x7f455c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.221+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f45640bc6c0 con 0x7f45700c4f50 2024-09-17T05:36:39.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.221+0000 7f456effd640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f455c078bc0 0x7f455c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.221+0000 7f456effd640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f455c078bc0 0x7f455c07b0d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4560005fd0 tx=0x7f4560005850 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.225+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f456403f030 con 0x7f45700c4f50 2024-09-17T05:36:39.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.404+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f45701ef9e0 con 0x7f455c078bc0 2024-09-17T05:36:39.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.405+0000 7f45577fe640 1 -- 172.21.15.117:0/2247750925 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f456000fad0 con 0x7f455c078bc0 2024-09-17T05:36:39.405 INFO:teuthology.orchestra.run.smithi117.stderr:instructed osd(s) 7 to deep-scrub 2024-09-17T05:36:39.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f455c078bc0 msgr2=0x7f455c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f455c078bc0 0x7f455c07b0d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4560005fd0 tx=0x7f4560005850 comp rx=0 tx=0).stop 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 msgr2=0x7f457006cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 0x7f457006cf80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4564009950 tx=0x7f4564033c90 comp rx=0 tx=0).stop 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 shutdown_connections 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f455c078bc0 0x7f455c07b0d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45700c67a0 0x7f4570145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45700c4f50 0x7f457006cf80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.407+0000 7f45752d5640 1 --2- 172.21.15.117:0/2247750925 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4570071850 0x7f45701efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.408+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 >> 172.21.15.117:0/2247750925 conn(0x7f457006e590 msgr2=0x7f45701481c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.408+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 shutdown_connections 2024-09-17T05:36:39.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.408+0000 7f45752d5640 1 -- 172.21.15.117:0/2247750925 wait complete. 2024-09-17T05:36:39.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 -- 172.21.15.117:0/206733353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 msgr2=0x7f936015d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 --2- 172.21.15.117:0/206733353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936015d220 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9360074aa0 tx=0x7f935402a4d0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 -- 172.21.15.117:0/206733353 shutdown_connections 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 --2- 172.21.15.117:0/206733353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936015d220 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 --2- 172.21.15.117:0/206733353 >> v2:172.21.15.119:3300/0 conn(0x7f93600cad30 0x7f93600c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 -- 172.21.15.117:0/206733353 >> 172.21.15.117:0/206733353 conn(0x7f9360076430 msgr2=0x7f9360072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:39.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 -- 172.21.15.117:0/206733353 shutdown_connections 2024-09-17T05:36:39.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.537+0000 7f9365772640 1 -- 172.21.15.117:0/206733353 wait complete. 2024-09-17T05:36:39.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.539+0000 7f9365772640 1 Processor -- start 2024-09-17T05:36:39.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.539+0000 7f9365772640 1 -- start start 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f9365772640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93600cad30 0x7f936015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f9365772640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936014e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f9365772640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9360159b30 0x7f936013eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f9365772640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9360159f40 con 0x7f9360153260 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f9365772640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93600cbc30 con 0x7f9360159b30 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f9365772640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f9360151a10 con 0x7f93600cad30 2024-09-17T05:36:39.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936014e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936014e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:35070/0 (socket says 172.21.15.117:35070) 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 -- 172.21.15.117:0/2150869465 learned_addr learned my addr 172.21.15.117:0/2150869465 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935effd640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93600cad30 0x7f936015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935f7fe640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9360159b30 0x7f936013eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 -- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93600cad30 msgr2=0x7f936015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93600cad30 0x7f936015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 -- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9360159b30 msgr2=0x7f936013eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9360159b30 0x7f936013eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.540+0000 7f935e7fc640 1 -- 172.21.15.117:0/2150869465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93601f3420 con 0x7f9360153260 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.541+0000 7f935e7fc640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936014e880 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9354034040 tx=0x7f93540039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.541+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935403e030 con 0x7f9360153260 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.541+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93540047a0 con 0x7f9360153260 2024-09-17T05:36:39.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.541+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936015a840 con 0x7f9360153260 2024-09-17T05:36:39.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.541+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354035040 con 0x7f9360153260 2024-09-17T05:36:39.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.541+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936015c0e0 con 0x7f9360153260 2024-09-17T05:36:39.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.542+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93600ca900 con 0x7f9360153260 2024-09-17T05:36:39.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.543+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9354041030 con 0x7f9360153260 2024-09-17T05:36:39.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.543+0000 7f93477fe640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f934c078b90 0x7f934c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:39.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.543+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f93540bbd30 con 0x7f9360153260 2024-09-17T05:36:39.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.543+0000 7f935effd640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f934c078b90 0x7f934c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:39.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.544+0000 7f935effd640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f934c078b90 0x7f934c07b0a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f935000bb50 tx=0x7f935000b2e0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:39.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.547+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9354083770 con 0x7f9360153260 2024-09-17T05:36:39.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.733+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9360151500 con 0x7f934c078b90 2024-09-17T05:36:39.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.739+0000 7f93477fe640 1 -- 172.21.15.117:0/2150869465 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334774 (secure 0 0 0) 0x7f9360151500 con 0x7f934c078b90 2024-09-17T05:36:39.744 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:36:39.746 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:36:39.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.742+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f934c078b90 msgr2=0x7f934c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.742+0000 7f9365772640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f934c078b90 0x7f934c07b0a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f935000bb50 tx=0x7f935000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.742+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 msgr2=0x7f936014e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:39.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.742+0000 7f9365772640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936014e880 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9354034040 tx=0x7f93540039e0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 shutdown_connections 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f934c078b90 0x7f934c07b0a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9360159b30 0x7f936013eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9360153260 0x7f936014e880 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 --2- 172.21.15.117:0/2150869465 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f93600cad30 0x7f936015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 >> 172.21.15.117:0/2150869465 conn(0x7f9360076430 msgr2=0x7f9360156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 shutdown_connections 2024-09-17T05:36:39.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:39.743+0000 7f9365772640 1 -- 172.21.15.117:0/2150869465 wait complete. 2024-09-17T05:36:39.755 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-17T05:36:39.531624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1708396181,"num_objects":653,"num_object_clones":0,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":0,"num_read":940,"num_read_kb":78368,"num_write":2624129,"num_write_kb":32805481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180459,"ondisk_log_size":180459,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":10186564,"kb_used_data":2337812,"kb_used_omap":12,"kb_used_meta":7846835,"kb_avail":744788156,"statfs":{"total":773094113280,"available":762663071744,"internally_reserved":0,"allocated":2393919488,"data_stored":4636549814,"data_compressed":251045257,"data_compressed_allocated":966746112,"data_compressed_original":3211038720,"omap_allocated":12751,"internal_metadata":8035159601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,3,10,20,22,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":89,"apply_latency_ms":89,"commit_latency_ns":89000000,"apply_latency_ns":89000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11029402348,"num_objects":-2646,"num_object_clones":0,"num_object_copies":-10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2646,"num_whiteouts":0,"num_read":16,"num_read_kb":33,"num_write":4077,"num_write_kb":7692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003898"},"pg_stats":[{"pgid":"2.1f","version":"30'18","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:45.149722+0000","last_change":"2024-09-17T05:27:25.653372+0000","last_active":"2024-09-17T05:35:45.149722+0000","last_peered":"2024-09-17T05:35:45.149722+0000","last_clean":"2024-09-17T05:35:45.149722+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:35:45.149722+0000","last_undegraded":"2024-09-17T05:35:45.149722+0000","last_fullsized":"2024-09-17T05:35:45.149722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":4124,"num_write":15,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:35.585637+0000","last_change":"2024-09-17T05:27:26.656660+0000","last_active":"2024-09-17T05:35:35.585637+0000","last_peered":"2024-09-17T05:35:35.585637+0000","last_clean":"2024-09-17T05:35:35.585637+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:35:35.585637+0000","last_undegraded":"2024-09-17T05:35:35.585637+0000","last_fullsized":"2024-09-17T05:35:35.585637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:32:31.588492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"31'5831","reported_seq":5878,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.601263+0000","last_change":"2024-09-17T05:27:30.686793+0000","last_active":"2024-09-17T05:36:34.601263+0000","last_peered":"2024-09-17T05:36:34.601263+0000","last_clean":"2024-09-17T05:36:34.601263+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:36:34.601263+0000","last_undegraded":"2024-09-17T05:36:34.601263+0000","last_fullsized":"2024-09-17T05:36:34.601263+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:57.901099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96587789,"num_objects":33,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":86628,"num_write_kb":1098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"29'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.956723+0000","last_change":"2024-09-17T05:27:25.653598+0000","last_active":"2024-09-17T05:35:30.956723+0000","last_peered":"2024-09-17T05:35:30.956723+0000","last_clean":"2024-09-17T05:35:30.956723+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:35:30.956723+0000","last_undegraded":"2024-09-17T05:35:30.956723+0000","last_fullsized":"2024-09-17T05:35:30.956723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:42.274030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":18,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:31.240343+0000","last_change":"2024-09-17T05:27:26.658996+0000","last_active":"2024-09-17T05:36:31.240343+0000","last_peered":"2024-09-17T05:36:31.240343+0000","last_clean":"2024-09-17T05:36:31.240343+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:36:31.240343+0000","last_undegraded":"2024-09-17T05:36:31.240343+0000","last_fullsized":"2024-09-17T05:36:31.240343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:21.297271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"31'5757","reported_seq":5805,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.358246+0000","last_change":"2024-09-17T05:27:30.678307+0000","last_active":"2024-09-17T05:36:35.358246+0000","last_peered":"2024-09-17T05:36:35.358246+0000","last_clean":"2024-09-17T05:36:35.358246+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:36:35.358246+0000","last_undegraded":"2024-09-17T05:36:35.358246+0000","last_fullsized":"2024-09-17T05:36:35.358246+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:41.513070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46163113,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":85086,"num_write_kb":1045061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"31'53","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.049914+0000","last_change":"2024-09-17T05:27:25.659960+0000","last_active":"2024-09-17T05:36:38.049914+0000","last_peered":"2024-09-17T05:36:38.049914+0000","last_clean":"2024-09-17T05:36:38.049914+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:36:38.049914+0000","last_undegraded":"2024-09-17T05:36:38.049914+0000","last_fullsized":"2024-09-17T05:36:38.049914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":8212,"num_write":40,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"28'7","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.704169+0000","last_change":"2024-09-17T05:36:35.704169+0000","last_active":"2024-09-17T05:36:35.704169+0000","last_peered":"2024-09-17T05:36:35.704169+0000","last_clean":"2024-09-17T05:36:35.704169+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:36:35.704169+0000","last_undegraded":"2024-09-17T05:36:35.704169+0000","last_fullsized":"2024-09-17T05:36:35.704169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'7","last_scrub_stamp":"2024-09-17T05:36:35.701723+0000","last_deep_scrub":"28'7","last_deep_scrub_stamp":"2024-09-17T05:36:35.701723+0000","last_clean_scrub_stamp":"2024-09-17T05:36:35.701723+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:41.922299+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'5370","reported_seq":5419,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.066908+0000","last_change":"2024-09-17T05:27:30.673784+0000","last_active":"2024-09-17T05:36:35.066908+0000","last_peered":"2024-09-17T05:36:35.066908+0000","last_clean":"2024-09-17T05:36:35.066908+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:36:35.066908+0000","last_undegraded":"2024-09-17T05:36:35.066908+0000","last_fullsized":"2024-09-17T05:36:35.066908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88097417,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79442,"num_write_kb":982831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"31'205","reported_seq":254,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.674515+0000","last_change":"2024-09-17T05:36:35.674394+0000","last_active":"2024-09-17T05:36:35.674515+0000","last_peered":"2024-09-17T05:36:35.674515+0000","last_clean":"2024-09-17T05:36:35.674515+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:36:35.674515+0000","last_undegraded":"2024-09-17T05:36:35.674515+0000","last_fullsized":"2024-09-17T05:36:35.674515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'204","last_scrub_stamp":"2024-09-17T05:36:35.674263+0000","last_deep_scrub":"31'204","last_deep_scrub_stamp":"2024-09-17T05:36:35.674263+0000","last_clean_scrub_stamp":"2024-09-17T05:36:35.674263+0000","objects_scrubbed":2,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:04.566769+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2480742,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":83,"num_write":202,"num_write_kb":2537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:31.414262+0000","last_change":"2024-09-17T05:27:26.662571+0000","last_active":"2024-09-17T05:36:31.414262+0000","last_peered":"2024-09-17T05:36:31.414262+0000","last_clean":"2024-09-17T05:36:31.414262+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:36:31.414262+0000","last_undegraded":"2024-09-17T05:36:31.414262+0000","last_fullsized":"2024-09-17T05:36:31.414262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:30.048596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"31'4609","reported_seq":4656,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.387706+0000","last_change":"2024-09-17T05:27:30.685784+0000","last_active":"2024-09-17T05:36:35.387706+0000","last_peered":"2024-09-17T05:36:35.387706+0000","last_clean":"2024-09-17T05:36:35.387706+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:36:35.387706+0000","last_undegraded":"2024-09-17T05:36:35.387706+0000","last_fullsized":"2024-09-17T05:36:35.387706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:12.578247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88104354,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":68233,"num_write_kb":860103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:14.605673+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:36:14.605673+0000","last_peered":"2024-09-17T05:36:14.605673+0000","last_clean":"2024-09-17T05:36:14.605673+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:36:14.605673+0000","last_undegraded":"2024-09-17T05:36:14.605673+0000","last_fullsized":"2024-09-17T05:36:14.605673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:35:36.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"31'126","reported_seq":169,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:19.838914+0000","last_change":"2024-09-17T05:27:25.662204+0000","last_active":"2024-09-17T05:36:19.838914+0000","last_peered":"2024-09-17T05:36:19.838914+0000","last_clean":"2024-09-17T05:36:19.838914+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:36:19.838914+0000","last_undegraded":"2024-09-17T05:36:19.838914+0000","last_fullsized":"2024-09-17T05:36:19.838914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:54.678965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4104,"num_write":119,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"31'5658","reported_seq":5700,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.030915+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:36:35.030915+0000","last_peered":"2024-09-17T05:36:35.030915+0000","last_clean":"2024-09-17T05:36:35.030915+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:36:35.030915+0000","last_undegraded":"2024-09-17T05:36:35.030915+0000","last_fullsized":"2024-09-17T05:36:35.030915+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:51.036493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67253812,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":83720,"num_write_kb":1023293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"31'6","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.933567+0000","last_change":"2024-09-17T05:27:26.661340+0000","last_active":"2024-09-17T05:36:24.933567+0000","last_peered":"2024-09-17T05:36:24.933567+0000","last_clean":"2024-09-17T05:36:24.933567+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:36:24.933567+0000","last_undegraded":"2024-09-17T05:36:24.933567+0000","last_fullsized":"2024-09-17T05:36:24.933567+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"31'6","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:34.663343+0000","last_change":"2024-09-17T05:27:25.656360+0000","last_active":"2024-09-17T05:35:34.663343+0000","last_peered":"2024-09-17T05:35:34.663343+0000","last_clean":"2024-09-17T05:35:34.663343+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:35:34.663343+0000","last_undegraded":"2024-09-17T05:35:34.663343+0000","last_fullsized":"2024-09-17T05:35:34.663343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"31'5558","reported_seq":5603,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.155527+0000","last_change":"2024-09-17T05:27:30.681008+0000","last_active":"2024-09-17T05:36:38.155527+0000","last_peered":"2024-09-17T05:36:38.155527+0000","last_clean":"2024-09-17T05:36:38.155527+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:36:38.155527+0000","last_undegraded":"2024-09-17T05:36:38.155527+0000","last_fullsized":"2024-09-17T05:36:38.155527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":81978,"num_write_kb":1031804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:09.528689+0000","last_change":"2024-09-17T05:27:26.655483+0000","last_active":"2024-09-17T05:36:09.528689+0000","last_peered":"2024-09-17T05:36:09.528689+0000","last_clean":"2024-09-17T05:36:09.528689+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:36:09.528689+0000","last_undegraded":"2024-09-17T05:36:09.528689+0000","last_fullsized":"2024-09-17T05:36:09.528689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:21.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"30'4","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953261+0000","last_change":"2024-09-17T05:27:25.656917+0000","last_active":"2024-09-17T05:35:25.953261+0000","last_peered":"2024-09-17T05:35:25.953261+0000","last_clean":"2024-09-17T05:35:25.953261+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:35:25.953261+0000","last_undegraded":"2024-09-17T05:35:25.953261+0000","last_fullsized":"2024-09-17T05:35:25.953261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.679982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'6547","reported_seq":6587,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.015360+0000","last_change":"2024-09-17T05:28:07.483827+0000","last_active":"2024-09-17T05:36:38.015360+0000","last_peered":"2024-09-17T05:36:38.015360+0000","last_clean":"2024-09-17T05:36:38.015360+0000","last_became_active":"2024-09-17T05:28:07.483263+0000","last_became_peered":"2024-09-17T05:28:07.483263+0000","last_unstale":"2024-09-17T05:36:38.015360+0000","last_undegraded":"2024-09-17T05:36:38.015360+0000","last_fullsized":"2024-09-17T05:36:38.015360+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":96660,"num_write_kb":1196264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2,3],"acting":[0,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:33.267368+0000","last_change":"2024-09-17T05:27:26.658943+0000","last_active":"2024-09-17T05:36:33.267368+0000","last_peered":"2024-09-17T05:36:33.267368+0000","last_clean":"2024-09-17T05:36:33.267368+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:36:33.267368+0000","last_undegraded":"2024-09-17T05:36:33.267368+0000","last_fullsized":"2024-09-17T05:36:33.267368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:18.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"31'17","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.014975+0000","last_change":"2024-09-17T05:36:34.672209+0000","last_active":"2024-09-17T05:36:38.014975+0000","last_peered":"2024-09-17T05:36:38.014975+0000","last_clean":"2024-09-17T05:36:38.014975+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:36:38.014975+0000","last_undegraded":"2024-09-17T05:36:38.014975+0000","last_fullsized":"2024-09-17T05:36:38.014975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:34.672070+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:34.672070+0000","last_clean_scrub_stamp":"2024-09-17T05:36:34.672070+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:12.516206+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":12,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"31'4835","reported_seq":4877,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.458756+0000","last_change":"2024-09-17T05:27:30.678131+0000","last_active":"2024-09-17T05:36:35.458756+0000","last_peered":"2024-09-17T05:36:35.458756+0000","last_clean":"2024-09-17T05:36:35.458756+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:36:35.458756+0000","last_undegraded":"2024-09-17T05:36:35.458756+0000","last_fullsized":"2024-09-17T05:36:35.458756+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50547365,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":71403,"num_write_kb":889518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'15","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:03.944697+0000","last_change":"2024-09-17T05:27:26.655634+0000","last_active":"2024-09-17T05:36:03.944697+0000","last_peered":"2024-09-17T05:36:03.944697+0000","last_clean":"2024-09-17T05:36:03.944697+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:36:03.944697+0000","last_undegraded":"2024-09-17T05:36:03.944697+0000","last_fullsized":"2024-09-17T05:36:03.944697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:59.087384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:04.068629+0000","last_change":"2024-09-17T05:27:25.656014+0000","last_active":"2024-09-17T05:36:04.068629+0000","last_peered":"2024-09-17T05:36:04.068629+0000","last_clean":"2024-09-17T05:36:04.068629+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:36:04.068629+0000","last_undegraded":"2024-09-17T05:36:04.068629+0000","last_fullsized":"2024-09-17T05:36:04.068629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:15.713596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"31'5530","reported_seq":5579,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.213333+0000","last_change":"2024-09-17T05:27:30.672177+0000","last_active":"2024-09-17T05:36:35.213333+0000","last_peered":"2024-09-17T05:36:35.213333+0000","last_clean":"2024-09-17T05:36:35.213333+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:36:35.213333+0000","last_undegraded":"2024-09-17T05:36:35.213333+0000","last_fullsized":"2024-09-17T05:36:35.213333+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5530,"log_dups_size":0,"ondisk_log_size":5530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:32:14.028441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58731580,"num_objects":17,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":81749,"num_write_kb":1039133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"31'11","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:09.567617+0000","last_change":"2024-09-17T05:27:26.654168+0000","last_active":"2024-09-17T05:36:09.567617+0000","last_peered":"2024-09-17T05:36:09.567617+0000","last_clean":"2024-09-17T05:36:09.567617+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:36:09.567617+0000","last_undegraded":"2024-09-17T05:36:09.567617+0000","last_fullsized":"2024-09-17T05:36:09.567617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:50.312518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"31'218","reported_seq":262,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.724158+0000","last_change":"2024-09-17T05:27:25.654032+0000","last_active":"2024-09-17T05:36:27.724158+0000","last_peered":"2024-09-17T05:36:27.724158+0000","last_clean":"2024-09-17T05:36:27.724158+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:36:27.724158+0000","last_undegraded":"2024-09-17T05:36:27.724158+0000","last_fullsized":"2024-09-17T05:36:27.724158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":21,"num_write":444,"num_write_kb":181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"31'6002","reported_seq":6053,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.893896+0000","last_change":"2024-09-17T05:27:30.672807+0000","last_active":"2024-09-17T05:36:34.893896+0000","last_peered":"2024-09-17T05:36:34.893896+0000","last_clean":"2024-09-17T05:36:34.893896+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:36:34.893896+0000","last_undegraded":"2024-09-17T05:36:34.893896+0000","last_fullsized":"2024-09-17T05:36:34.893896+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:19.254198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62929263,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":88874,"num_write_kb":1123429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.338375+0000","last_change":"2024-09-17T05:27:26.658306+0000","last_active":"2024-09-17T05:36:27.338375+0000","last_peered":"2024-09-17T05:36:27.338375+0000","last_clean":"2024-09-17T05:36:27.338375+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:36:27.338375+0000","last_undegraded":"2024-09-17T05:36:27.338375+0000","last_fullsized":"2024-09-17T05:36:27.338375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:12.519083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"25'1","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.066159+0000","last_change":"2024-09-17T05:27:25.646755+0000","last_active":"2024-09-17T05:35:26.066159+0000","last_peered":"2024-09-17T05:35:26.066159+0000","last_clean":"2024-09-17T05:35:26.066159+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:35:26.066159+0000","last_undegraded":"2024-09-17T05:35:26.066159+0000","last_fullsized":"2024-09-17T05:35:26.066159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.665327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'5074","reported_seq":5116,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.708951+0000","last_change":"2024-09-17T05:27:30.677733+0000","last_active":"2024-09-17T05:36:37.708951+0000","last_peered":"2024-09-17T05:36:37.708951+0000","last_clean":"2024-09-17T05:36:37.708951+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:36:37.708951+0000","last_undegraded":"2024-09-17T05:36:37.708951+0000","last_fullsized":"2024-09-17T05:36:37.708951+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":75016,"num_write_kb":939196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"31'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.951700+0000","last_change":"2024-09-17T05:27:26.655255+0000","last_active":"2024-09-17T05:36:25.951700+0000","last_peered":"2024-09-17T05:36:25.951700+0000","last_clean":"2024-09-17T05:36:25.951700+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:36:25.951700+0000","last_undegraded":"2024-09-17T05:36:25.951700+0000","last_fullsized":"2024-09-17T05:36:25.951700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:06.671506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"28'1","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.968299+0000","last_change":"2024-09-17T05:27:25.652576+0000","last_active":"2024-09-17T05:35:25.968299+0000","last_peered":"2024-09-17T05:35:25.968299+0000","last_clean":"2024-09-17T05:35:25.968299+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:35:25.968299+0000","last_undegraded":"2024-09-17T05:35:25.968299+0000","last_fullsized":"2024-09-17T05:35:25.968299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:47.242392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"31'6221","reported_seq":6269,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.755564+0000","last_change":"2024-09-17T05:27:30.687640+0000","last_active":"2024-09-17T05:36:37.755564+0000","last_peered":"2024-09-17T05:36:37.755564+0000","last_clean":"2024-09-17T05:36:37.755564+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:36:37.755564+0000","last_undegraded":"2024-09-17T05:36:37.755564+0000","last_fullsized":"2024-09-17T05:36:37.755564+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18859,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":92052,"num_write_kb":1145505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.046100+0000","last_change":"2024-09-17T05:27:26.658730+0000","last_active":"2024-09-17T05:36:38.046100+0000","last_peered":"2024-09-17T05:36:38.046100+0000","last_clean":"2024-09-17T05:36:38.046100+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:36:38.046100+0000","last_undegraded":"2024-09-17T05:36:38.046100+0000","last_fullsized":"2024-09-17T05:36:38.046100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"30'41","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.332894+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:35:32.332894+0000","last_peered":"2024-09-17T05:35:32.332894+0000","last_clean":"2024-09-17T05:35:32.332894+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:35:32.332894+0000","last_undegraded":"2024-09-17T05:35:32.332894+0000","last_fullsized":"2024-09-17T05:35:32.332894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:59:20.808704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":35,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'5354","reported_seq":5404,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.691545+0000","last_change":"2024-09-17T05:27:30.686379+0000","last_active":"2024-09-17T05:36:37.691545+0000","last_peered":"2024-09-17T05:36:37.691545+0000","last_clean":"2024-09-17T05:36:37.691545+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:36:37.691545+0000","last_undegraded":"2024-09-17T05:36:37.691545+0000","last_fullsized":"2024-09-17T05:36:37.691545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5354,"log_dups_size":0,"ondisk_log_size":5354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76947,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79027,"num_write_kb":985615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"28'11","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949408+0000","last_change":"2024-09-17T05:27:26.656598+0000","last_active":"2024-09-17T05:35:25.949408+0000","last_peered":"2024-09-17T05:35:25.949408+0000","last_clean":"2024-09-17T05:35:25.949408+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:35:25.949408+0000","last_undegraded":"2024-09-17T05:35:25.949408+0000","last_fullsized":"2024-09-17T05:35:25.949408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:13.405988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"31'30","reported_seq":91,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.768128+0000","last_change":"2024-09-17T05:27:25.656104+0000","last_active":"2024-09-17T05:36:28.768128+0000","last_peered":"2024-09-17T05:36:28.768128+0000","last_clean":"2024-09-17T05:36:28.768128+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:36:28.768128+0000","last_undegraded":"2024-09-17T05:36:28.768128+0000","last_fullsized":"2024-09-17T05:36:28.768128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94359,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":116,"num_write":27,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'5922","reported_seq":5973,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.364047+0000","last_change":"2024-09-17T05:27:30.678120+0000","last_active":"2024-09-17T05:36:37.364047+0000","last_peered":"2024-09-17T05:36:37.364047+0000","last_clean":"2024-09-17T05:36:37.364047+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:36:37.364047+0000","last_undegraded":"2024-09-17T05:36:37.364047+0000","last_fullsized":"2024-09-17T05:36:37.364047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:31.236171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53191,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":87388,"num_write_kb":1093347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"31'5","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:44.954006+0000","last_change":"2024-09-17T05:27:26.653927+0000","last_active":"2024-09-17T05:35:44.954006+0000","last_peered":"2024-09-17T05:35:44.954006+0000","last_clean":"2024-09-17T05:35:44.954006+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:35:44.954006+0000","last_undegraded":"2024-09-17T05:35:44.954006+0000","last_fullsized":"2024-09-17T05:35:44.954006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:09.547400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"30'91","reported_seq":136,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.303333+0000","last_change":"2024-09-17T05:27:25.658631+0000","last_active":"2024-09-17T05:35:31.303333+0000","last_peered":"2024-09-17T05:35:31.303333+0000","last_clean":"2024-09-17T05:35:31.303333+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:35:31.303333+0000","last_undegraded":"2024-09-17T05:35:31.303333+0000","last_fullsized":"2024-09-17T05:35:31.303333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.874933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8204,"num_write":88,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"31'4816","reported_seq":4858,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.083898+0000","last_change":"2024-09-17T05:27:30.689719+0000","last_active":"2024-09-17T05:36:37.083898+0000","last_peered":"2024-09-17T05:36:37.083898+0000","last_clean":"2024-09-17T05:36:37.083898+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:36:37.083898+0000","last_undegraded":"2024-09-17T05:36:37.083898+0000","last_fullsized":"2024-09-17T05:36:37.083898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:18.584873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13015874,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":71010,"num_write_kb":889582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"31'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:33.066415+0000","last_change":"2024-09-17T05:27:26.652981+0000","last_active":"2024-09-17T05:36:33.066415+0000","last_peered":"2024-09-17T05:36:33.066415+0000","last_clean":"2024-09-17T05:36:33.066415+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:36:33.066415+0000","last_undegraded":"2024-09-17T05:36:33.066415+0000","last_fullsized":"2024-09-17T05:36:33.066415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:31.451890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"30'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.201104+0000","last_change":"2024-09-17T05:27:25.662777+0000","last_active":"2024-09-17T05:35:26.201104+0000","last_peered":"2024-09-17T05:35:26.201104+0000","last_clean":"2024-09-17T05:35:26.201104+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:35:26.201104+0000","last_undegraded":"2024-09-17T05:35:26.201104+0000","last_fullsized":"2024-09-17T05:35:26.201104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:22.785750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":135,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.187694+0000","last_change":"2024-09-17T05:27:15.488331+0000","last_active":"2024-09-17T05:35:26.187694+0000","last_peered":"2024-09-17T05:35:26.187694+0000","last_clean":"2024-09-17T05:35:26.187694+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:35:26.187694+0000","last_undegraded":"2024-09-17T05:35:26.187694+0000","last_fullsized":"2024-09-17T05:35:26.187694+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:14.119421+0000","last_clean_scrub_stamp":"2024-09-17T05:27:14.119421+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:15.665966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'5394","reported_seq":5443,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.934152+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:36:34.934152+0000","last_peered":"2024-09-17T05:36:34.934152+0000","last_clean":"2024-09-17T05:36:34.934152+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:36:34.934152+0000","last_undegraded":"2024-09-17T05:36:34.934152+0000","last_fullsized":"2024-09-17T05:36:34.934152+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:03.905142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79701508,"num_objects":24,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":80025,"num_write_kb":1015697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"31'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.092338+0000","last_change":"2024-09-17T05:27:26.662935+0000","last_active":"2024-09-17T05:36:34.092338+0000","last_peered":"2024-09-17T05:36:34.092338+0000","last_clean":"2024-09-17T05:36:34.092338+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:36:34.092338+0000","last_undegraded":"2024-09-17T05:36:34.092338+0000","last_fullsized":"2024-09-17T05:36:34.092338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:13.067504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"30'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.981638+0000","last_change":"2024-09-17T05:27:25.661124+0000","last_active":"2024-09-17T05:35:25.981638+0000","last_peered":"2024-09-17T05:35:25.981638+0000","last_clean":"2024-09-17T05:35:25.981638+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:35:25.981638+0000","last_undegraded":"2024-09-17T05:35:25.981638+0000","last_fullsized":"2024-09-17T05:35:25.981638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"31'5560","reported_seq":5608,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.160007+0000","last_change":"2024-09-17T05:27:30.678075+0000","last_active":"2024-09-17T05:36:35.160007+0000","last_peered":"2024-09-17T05:36:35.160007+0000","last_clean":"2024-09-17T05:36:35.160007+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:36:35.160007+0000","last_undegraded":"2024-09-17T05:36:35.160007+0000","last_fullsized":"2024-09-17T05:36:35.160007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:25.764473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58722217,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82300,"num_write_kb":1025442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'14","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.139906+0000","last_change":"2024-09-17T05:36:36.597250+0000","last_active":"2024-09-17T05:36:38.139906+0000","last_peered":"2024-09-17T05:36:38.139906+0000","last_clean":"2024-09-17T05:36:38.139906+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:36:38.139906+0000","last_undegraded":"2024-09-17T05:36:38.139906+0000","last_fullsized":"2024-09-17T05:36:38.139906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:36.597064+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:36.597064+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.597064+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:09.330050+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"30'20","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:32.072706+0000","last_change":"2024-09-17T05:27:25.653217+0000","last_active":"2024-09-17T05:35:32.072706+0000","last_peered":"2024-09-17T05:35:32.072706+0000","last_clean":"2024-09-17T05:35:32.072706+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:35:32.072706+0000","last_undegraded":"2024-09-17T05:35:32.072706+0000","last_fullsized":"2024-09-17T05:35:32.072706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:23:44.786863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":19,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"31'5075","reported_seq":5124,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.201051+0000","last_change":"2024-09-17T05:27:30.680006+0000","last_active":"2024-09-17T05:36:37.201051+0000","last_peered":"2024-09-17T05:36:37.201051+0000","last_clean":"2024-09-17T05:36:37.201051+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:36:37.201051+0000","last_undegraded":"2024-09-17T05:36:37.201051+0000","last_fullsized":"2024-09-17T05:36:37.201051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:31.245209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8414516,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":74955,"num_write_kb":933970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"31'12","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.028643+0000","last_change":"2024-09-17T05:27:26.663005+0000","last_active":"2024-09-17T05:36:38.028643+0000","last_peered":"2024-09-17T05:36:38.028643+0000","last_clean":"2024-09-17T05:36:38.028643+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:36:38.028643+0000","last_undegraded":"2024-09-17T05:36:38.028643+0000","last_fullsized":"2024-09-17T05:36:38.028643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'27","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.843028+0000","last_change":"2024-09-17T05:27:25.651323+0000","last_active":"2024-09-17T05:35:50.843028+0000","last_peered":"2024-09-17T05:35:50.843028+0000","last_clean":"2024-09-17T05:35:50.843028+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:35:50.843028+0000","last_undegraded":"2024-09-17T05:35:50.843028+0000","last_fullsized":"2024-09-17T05:35:50.843028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:32.391558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":26,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"31'5154","reported_seq":5199,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.663647+0000","last_change":"2024-09-17T05:27:30.682368+0000","last_active":"2024-09-17T05:36:34.663647+0000","last_peered":"2024-09-17T05:36:34.663647+0000","last_clean":"2024-09-17T05:36:34.663647+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:36:34.663647+0000","last_undegraded":"2024-09-17T05:36:34.663647+0000","last_fullsized":"2024-09-17T05:36:34.663647+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:48:14.091592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96489082,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76447,"num_write_kb":955028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"31'5","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:33.450662+0000","last_change":"2024-09-17T05:27:26.661584+0000","last_active":"2024-09-17T05:36:33.450662+0000","last_peered":"2024-09-17T05:36:33.450662+0000","last_clean":"2024-09-17T05:36:33.450662+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:36:33.450662+0000","last_undegraded":"2024-09-17T05:36:33.450662+0000","last_fullsized":"2024-09-17T05:36:33.450662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.855486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"31'11","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.125445+0000","last_change":"2024-09-17T05:27:25.658314+0000","last_active":"2024-09-17T05:36:38.125445+0000","last_peered":"2024-09-17T05:36:38.125445+0000","last_clean":"2024-09-17T05:36:38.125445+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:36:38.125445+0000","last_undegraded":"2024-09-17T05:36:38.125445+0000","last_fullsized":"2024-09-17T05:36:38.125445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"31'5191","reported_seq":5236,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:36.344644+0000","last_change":"2024-09-17T05:27:30.678741+0000","last_active":"2024-09-17T05:36:36.344644+0000","last_peered":"2024-09-17T05:36:36.344644+0000","last_clean":"2024-09-17T05:36:36.344644+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:36:36.344644+0000","last_undegraded":"2024-09-17T05:36:36.344644+0000","last_fullsized":"2024-09-17T05:36:36.344644+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5191,"log_dups_size":0,"ondisk_log_size":5191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16856525,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76679,"num_write_kb":970760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.073783+0000","last_change":"2024-09-17T05:27:26.659277+0000","last_active":"2024-09-17T05:35:26.073783+0000","last_peered":"2024-09-17T05:35:26.073783+0000","last_clean":"2024-09-17T05:35:26.073783+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:35:26.073783+0000","last_undegraded":"2024-09-17T05:35:26.073783+0000","last_fullsized":"2024-09-17T05:35:26.073783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:52.738891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"30'19","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.833437+0000","last_change":"2024-09-17T05:27:25.653937+0000","last_active":"2024-09-17T05:35:50.833437+0000","last_peered":"2024-09-17T05:35:50.833437+0000","last_clean":"2024-09-17T05:35:50.833437+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:35:50.833437+0000","last_undegraded":"2024-09-17T05:35:50.833437+0000","last_fullsized":"2024-09-17T05:35:50.833437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:45.105323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":40,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"31'5817","reported_seq":5868,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.093799+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:36:37.093799+0000","last_peered":"2024-09-17T05:36:37.093799+0000","last_clean":"2024-09-17T05:36:37.093799+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:36:37.093799+0000","last_undegraded":"2024-09-17T05:36:37.093799+0000","last_fullsized":"2024-09-17T05:36:37.093799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253729,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":85907,"num_write_kb":1086559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.039122+0000","last_change":"2024-09-17T05:27:26.657916+0000","last_active":"2024-09-17T05:36:34.039122+0000","last_peered":"2024-09-17T05:36:34.039122+0000","last_clean":"2024-09-17T05:36:34.039122+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:36:34.039122+0000","last_undegraded":"2024-09-17T05:36:34.039122+0000","last_fullsized":"2024-09-17T05:36:34.039122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"31'17","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.104082+0000","last_change":"2024-09-17T05:27:25.657534+0000","last_active":"2024-09-17T05:36:38.104082+0000","last_peered":"2024-09-17T05:36:38.104082+0000","last_clean":"2024-09-17T05:36:38.104082+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:36:38.104082+0000","last_undegraded":"2024-09-17T05:36:38.104082+0000","last_fullsized":"2024-09-17T05:36:38.104082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":14,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"31'5914","reported_seq":5962,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.165157+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:36:37.165157+0000","last_peered":"2024-09-17T05:36:37.165157+0000","last_clean":"2024-09-17T05:36:37.165157+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:36:37.165157+0000","last_undegraded":"2024-09-17T05:36:37.165157+0000","last_fullsized":"2024-09-17T05:36:37.165157+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25225446,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87422,"num_write_kb":1070021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"28'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.200625+0000","last_change":"2024-09-17T05:27:26.665669+0000","last_active":"2024-09-17T05:35:26.200625+0000","last_peered":"2024-09-17T05:35:26.200625+0000","last_clean":"2024-09-17T05:35:26.200625+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:35:26.200625+0000","last_undegraded":"2024-09-17T05:35:26.200625+0000","last_fullsized":"2024-09-17T05:35:26.200625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:29.081426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"30'6","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:26.335841+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:35:26.335841+0000","last_peered":"2024-09-17T05:35:26.335841+0000","last_clean":"2024-09-17T05:35:26.335841+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:35:26.335841+0000","last_undegraded":"2024-09-17T05:35:26.335841+0000","last_fullsized":"2024-09-17T05:35:26.335841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:46.257303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'5939","reported_seq":5987,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.344314+0000","last_change":"2024-09-17T05:27:30.687591+0000","last_active":"2024-09-17T05:36:37.344314+0000","last_peered":"2024-09-17T05:36:37.344314+0000","last_clean":"2024-09-17T05:36:37.344314+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:36:37.344314+0000","last_undegraded":"2024-09-17T05:36:37.344314+0000","last_fullsized":"2024-09-17T05:36:37.344314+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5939,"log_dups_size":0,"ondisk_log_size":5939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:10:35.887344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232139,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87602,"num_write_kb":1085839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"31'14","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.121519+0000","last_change":"2024-09-17T05:27:26.661899+0000","last_active":"2024-09-17T05:36:35.121519+0000","last_peered":"2024-09-17T05:36:35.121519+0000","last_clean":"2024-09-17T05:36:35.121519+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:36:35.121519+0000","last_undegraded":"2024-09-17T05:36:35.121519+0000","last_fullsized":"2024-09-17T05:36:35.121519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:24.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"30'87","reported_seq":134,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.501205+0000","last_change":"2024-09-17T05:27:25.660974+0000","last_active":"2024-09-17T05:35:31.501205+0000","last_peered":"2024-09-17T05:35:31.501205+0000","last_clean":"2024-09-17T05:35:31.501205+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:35:31.501205+0000","last_undegraded":"2024-09-17T05:35:31.501205+0000","last_fullsized":"2024-09-17T05:35:31.501205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:55.344964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":8218,"num_write":88,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"31'5207","reported_seq":5253,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.030734+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:36:35.030734+0000","last_peered":"2024-09-17T05:36:35.030734+0000","last_clean":"2024-09-17T05:36:35.030734+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:36:35.030734+0000","last_undegraded":"2024-09-17T05:36:35.030734+0000","last_fullsized":"2024-09-17T05:36:35.030734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:27.087162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":76970,"num_write_kb":960359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"31'11","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:30.623855+0000","last_change":"2024-09-17T05:27:26.656768+0000","last_active":"2024-09-17T05:36:30.623855+0000","last_peered":"2024-09-17T05:36:30.623855+0000","last_clean":"2024-09-17T05:36:30.623855+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:36:30.623855+0000","last_undegraded":"2024-09-17T05:36:30.623855+0000","last_fullsized":"2024-09-17T05:36:30.623855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"31'214","reported_seq":259,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.689738+0000","last_change":"2024-09-17T05:27:25.662319+0000","last_active":"2024-09-17T05:36:24.689738+0000","last_peered":"2024-09-17T05:36:24.689738+0000","last_clean":"2024-09-17T05:36:24.689738+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:36:24.689738+0000","last_undegraded":"2024-09-17T05:36:24.689738+0000","last_fullsized":"2024-09-17T05:36:24.689738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:42.752587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":213,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"31'5394","reported_seq":5443,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.124312+0000","last_change":"2024-09-17T05:27:30.679702+0000","last_active":"2024-09-17T05:36:35.124312+0000","last_peered":"2024-09-17T05:36:35.124312+0000","last_clean":"2024-09-17T05:36:35.124312+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:36:35.124312+0000","last_undegraded":"2024-09-17T05:36:35.124312+0000","last_fullsized":"2024-09-17T05:36:35.124312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5394,"log_dups_size":0,"ondisk_log_size":5394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:56.762330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92357474,"num_objects":27,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79566,"num_write_kb":1008039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.953082+0000","last_change":"2024-09-17T05:27:26.653935+0000","last_active":"2024-09-17T05:36:28.953082+0000","last_peered":"2024-09-17T05:36:28.953082+0000","last_clean":"2024-09-17T05:36:28.953082+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:36:28.953082+0000","last_undegraded":"2024-09-17T05:36:28.953082+0000","last_fullsized":"2024-09-17T05:36:28.953082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:12:04.311075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"31'10","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:50.839506+0000","last_change":"2024-09-17T05:27:25.658516+0000","last_active":"2024-09-17T05:35:50.839506+0000","last_peered":"2024-09-17T05:35:50.839506+0000","last_clean":"2024-09-17T05:35:50.839506+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:35:50.839506+0000","last_undegraded":"2024-09-17T05:35:50.839506+0000","last_fullsized":"2024-09-17T05:35:50.839506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:08.026787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"31'5664","reported_seq":5709,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.426029+0000","last_change":"2024-09-17T05:27:30.678660+0000","last_active":"2024-09-17T05:36:35.426029+0000","last_peered":"2024-09-17T05:36:35.426029+0000","last_clean":"2024-09-17T05:36:35.426029+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:36:35.426029+0000","last_undegraded":"2024-09-17T05:36:35.426029+0000","last_fullsized":"2024-09-17T05:36:35.426029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:30.110749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54633457,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15,"num_read_kb":0,"num_write":83458,"num_write_kb":1022097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'11","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:31.065805+0000","last_change":"2024-09-17T05:27:26.661818+0000","last_active":"2024-09-17T05:36:31.065805+0000","last_peered":"2024-09-17T05:36:31.065805+0000","last_clean":"2024-09-17T05:36:31.065805+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:36:31.065805+0000","last_undegraded":"2024-09-17T05:36:31.065805+0000","last_fullsized":"2024-09-17T05:36:31.065805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:10.456933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"31'5","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.119286+0000","last_change":"2024-09-17T05:27:25.659039+0000","last_active":"2024-09-17T05:36:38.119286+0000","last_peered":"2024-09-17T05:36:38.119286+0000","last_clean":"2024-09-17T05:36:38.119286+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:36:38.119286+0000","last_undegraded":"2024-09-17T05:36:38.119286+0000","last_fullsized":"2024-09-17T05:36:38.119286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"31'5375","reported_seq":5418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.746718+0000","last_change":"2024-09-17T05:27:30.689366+0000","last_active":"2024-09-17T05:36:34.746718+0000","last_peered":"2024-09-17T05:36:34.746718+0000","last_clean":"2024-09-17T05:36:34.746718+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:36:34.746718+0000","last_undegraded":"2024-09-17T05:36:34.746718+0000","last_fullsized":"2024-09-17T05:36:34.746718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:16.542522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96537472,"num_objects":30,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":13,"num_read_kb":0,"num_write":79695,"num_write_kb":1002963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"31'7","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:30.872500+0000","last_change":"2024-09-17T05:27:26.652978+0000","last_active":"2024-09-17T05:36:30.872500+0000","last_peered":"2024-09-17T05:36:30.872500+0000","last_clean":"2024-09-17T05:36:30.872500+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:36:30.872500+0000","last_undegraded":"2024-09-17T05:36:30.872500+0000","last_fullsized":"2024-09-17T05:36:30.872500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:23.791928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"31'11","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.972204+0000","last_change":"2024-09-17T05:27:25.650695+0000","last_active":"2024-09-17T05:35:25.972204+0000","last_peered":"2024-09-17T05:35:25.972204+0000","last_clean":"2024-09-17T05:35:25.972204+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:35:25.972204+0000","last_undegraded":"2024-09-17T05:35:25.972204+0000","last_fullsized":"2024-09-17T05:35:25.972204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:30:18.275790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'5587","reported_seq":5634,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.332666+0000","last_change":"2024-09-17T05:27:30.679362+0000","last_active":"2024-09-17T05:36:35.332666+0000","last_peered":"2024-09-17T05:36:35.332666+0000","last_clean":"2024-09-17T05:36:35.332666+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:36:35.332666+0000","last_undegraded":"2024-09-17T05:36:35.332666+0000","last_fullsized":"2024-09-17T05:36:35.332666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:22.496212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67139217,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82691,"num_write_kb":1020969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6458","reported_seq":6506,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.371152+0000","last_change":"2024-09-17T05:27:30.677570+0000","last_active":"2024-09-17T05:36:35.371152+0000","last_peered":"2024-09-17T05:36:35.371152+0000","last_clean":"2024-09-17T05:36:35.371152+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:36:35.371152+0000","last_undegraded":"2024-09-17T05:36:35.371152+0000","last_fullsized":"2024-09-17T05:36:35.371152+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:59.261149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121646218,"num_objects":32,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":95921,"num_write_kb":1202164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.953061+0000","last_change":"2024-09-17T05:27:26.653903+0000","last_active":"2024-09-17T05:35:25.953061+0000","last_peered":"2024-09-17T05:35:25.953061+0000","last_clean":"2024-09-17T05:35:25.953061+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:35:25.953061+0000","last_undegraded":"2024-09-17T05:35:25.953061+0000","last_fullsized":"2024-09-17T05:35:25.953061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"30'27","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:31.703337+0000","last_change":"2024-09-17T05:27:25.650025+0000","last_active":"2024-09-17T05:35:31.703337+0000","last_peered":"2024-09-17T05:35:31.703337+0000","last_clean":"2024-09-17T05:35:31.703337+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:35:31.703337+0000","last_undegraded":"2024-09-17T05:35:31.703337+0000","last_fullsized":"2024-09-17T05:35:31.703337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:57.222429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":24,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"31'5598","reported_seq":5650,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.823925+0000","last_change":"2024-09-17T05:27:30.683958+0000","last_active":"2024-09-17T05:36:34.823925+0000","last_peered":"2024-09-17T05:36:34.823925+0000","last_clean":"2024-09-17T05:36:34.823925+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:36:34.823925+0000","last_undegraded":"2024-09-17T05:36:34.823925+0000","last_fullsized":"2024-09-17T05:36:34.823925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:53.399159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121717964,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":83022,"num_write_kb":1044536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"31'16","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.812277+0000","last_change":"2024-09-17T05:27:26.653257+0000","last_active":"2024-09-17T05:36:28.812277+0000","last_peered":"2024-09-17T05:36:28.812277+0000","last_clean":"2024-09-17T05:36:28.812277+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:36:28.812277+0000","last_undegraded":"2024-09-17T05:36:28.812277+0000","last_fullsized":"2024-09-17T05:36:28.812277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:15.393484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"30'5","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.962325+0000","last_change":"2024-09-17T05:27:25.660083+0000","last_active":"2024-09-17T05:35:25.962325+0000","last_peered":"2024-09-17T05:35:25.962325+0000","last_clean":"2024-09-17T05:35:25.962325+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:35:25.962325+0000","last_undegraded":"2024-09-17T05:35:25.962325+0000","last_fullsized":"2024-09-17T05:35:25.962325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.686328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"31'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:30.941350+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:35:30.941350+0000","last_peered":"2024-09-17T05:35:30.941350+0000","last_clean":"2024-09-17T05:35:30.941350+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:35:30.941350+0000","last_undegraded":"2024-09-17T05:35:30.941350+0000","last_fullsized":"2024-09-17T05:35:30.941350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:29.849308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":4108,"num_write":17,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.941816+0000","last_change":"2024-09-17T05:27:26.663253+0000","last_active":"2024-09-17T05:35:25.941816+0000","last_peered":"2024-09-17T05:35:25.941816+0000","last_clean":"2024-09-17T05:35:25.941816+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:35:25.941816+0000","last_undegraded":"2024-09-17T05:35:25.941816+0000","last_fullsized":"2024-09-17T05:35:25.941816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:25.498356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4598","reported_seq":4636,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.388371+0000","last_change":"2024-09-17T05:27:30.680844+0000","last_active":"2024-09-17T05:36:35.388371+0000","last_peered":"2024-09-17T05:36:35.388371+0000","last_clean":"2024-09-17T05:36:35.388371+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:36:35.388371+0000","last_undegraded":"2024-09-17T05:36:35.388371+0000","last_fullsized":"2024-09-17T05:36:35.388371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:38.034644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41944767,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":68102,"num_write_kb":833128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"30'1234","reported_seq":1275,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.957097+0000","last_change":"2024-09-17T05:27:25.654752+0000","last_active":"2024-09-17T05:35:25.957097+0000","last_peered":"2024-09-17T05:35:25.957097+0000","last_clean":"2024-09-17T05:35:25.957097+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:35:25.957097+0000","last_undegraded":"2024-09-17T05:35:25.957097+0000","last_fullsized":"2024-09-17T05:35:25.957097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":1234,"log_dups_size":0,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:46.202865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4196027,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1229,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:31.697351+0000","last_change":"2024-09-17T05:27:26.657949+0000","last_active":"2024-09-17T05:36:31.697351+0000","last_peered":"2024-09-17T05:36:31.697351+0000","last_clean":"2024-09-17T05:36:31.697351+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:36:31.697351+0000","last_undegraded":"2024-09-17T05:36:31.697351+0000","last_fullsized":"2024-09-17T05:36:31.697351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:12.612911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'6213","reported_seq":6262,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.809594+0000","last_change":"2024-09-17T05:27:30.684657+0000","last_active":"2024-09-17T05:36:34.809594+0000","last_peered":"2024-09-17T05:36:34.809594+0000","last_clean":"2024-09-17T05:36:34.809594+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:36:34.809594+0000","last_undegraded":"2024-09-17T05:36:34.809594+0000","last_fullsized":"2024-09-17T05:36:34.809594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":6213,"log_dups_size":0,"ondisk_log_size":6213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.261780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92380396,"num_objects":29,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":92272,"num_write_kb":1140422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"31'11","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:32.041706+0000","last_change":"2024-09-17T05:27:26.650086+0000","last_active":"2024-09-17T05:36:32.041706+0000","last_peered":"2024-09-17T05:36:32.041706+0000","last_clean":"2024-09-17T05:36:32.041706+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:36:32.041706+0000","last_undegraded":"2024-09-17T05:36:32.041706+0000","last_fullsized":"2024-09-17T05:36:32.041706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:58.228608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"31'13","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:29.397229+0000","last_change":"2024-09-17T05:27:26.652320+0000","last_active":"2024-09-17T05:36:29.397229+0000","last_peered":"2024-09-17T05:36:29.397229+0000","last_clean":"2024-09-17T05:36:29.397229+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:36:29.397229+0000","last_undegraded":"2024-09-17T05:36:29.397229+0000","last_fullsized":"2024-09-17T05:36:29.397229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"31'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:31.101874+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:36:31.101874+0000","last_peered":"2024-09-17T05:36:31.101874+0000","last_clean":"2024-09-17T05:36:31.101874+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:36:31.101874+0000","last_undegraded":"2024-09-17T05:36:31.101874+0000","last_fullsized":"2024-09-17T05:36:31.101874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:36:34.903607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:55.876152+0000","last_change":"2024-09-17T05:27:26.660278+0000","last_active":"2024-09-17T05:35:55.876152+0000","last_peered":"2024-09-17T05:35:55.876152+0000","last_clean":"2024-09-17T05:35:55.876152+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:35:55.876152+0000","last_undegraded":"2024-09-17T05:35:55.876152+0000","last_fullsized":"2024-09-17T05:35:55.876152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:06.332415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:06.559514+0000","last_change":"2024-09-17T05:27:26.661588+0000","last_active":"2024-09-17T05:36:06.559514+0000","last_peered":"2024-09-17T05:36:06.559514+0000","last_clean":"2024-09-17T05:36:06.559514+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:36:06.559514+0000","last_undegraded":"2024-09-17T05:36:06.559514+0000","last_fullsized":"2024-09-17T05:36:06.559514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.516908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.293785+0000","last_change":"2024-09-17T05:27:26.656526+0000","last_active":"2024-09-17T05:36:26.293785+0000","last_peered":"2024-09-17T05:36:26.293785+0000","last_clean":"2024-09-17T05:36:26.293785+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:36:26.293785+0000","last_undegraded":"2024-09-17T05:36:26.293785+0000","last_fullsized":"2024-09-17T05:36:26.293785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:36.238342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"31'6","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:26.437545+0000","last_change":"2024-09-17T05:27:26.656272+0000","last_active":"2024-09-17T05:36:26.437545+0000","last_peered":"2024-09-17T05:36:26.437545+0000","last_clean":"2024-09-17T05:36:26.437545+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:36:26.437545+0000","last_undegraded":"2024-09-17T05:36:26.437545+0000","last_fullsized":"2024-09-17T05:36:26.437545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:17.030561+0000","last_change":"2024-09-17T05:27:26.659425+0000","last_active":"2024-09-17T05:36:17.030561+0000","last_peered":"2024-09-17T05:36:17.030561+0000","last_clean":"2024-09-17T05:36:17.030561+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:36:17.030561+0000","last_undegraded":"2024-09-17T05:36:17.030561+0000","last_fullsized":"2024-09-17T05:36:17.030561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:25.496047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"31'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:27.222346+0000","last_change":"2024-09-17T05:27:26.664138+0000","last_active":"2024-09-17T05:36:27.222346+0000","last_peered":"2024-09-17T05:36:27.222346+0000","last_clean":"2024-09-17T05:36:27.222346+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:36:27.222346+0000","last_undegraded":"2024-09-17T05:36:27.222346+0000","last_fullsized":"2024-09-17T05:36:27.222346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:38.585574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"31'14","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:28.129905+0000","last_change":"2024-09-17T05:27:26.655630+0000","last_active":"2024-09-17T05:36:28.129905+0000","last_peered":"2024-09-17T05:36:28.129905+0000","last_clean":"2024-09-17T05:36:28.129905+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:36:28.129905+0000","last_undegraded":"2024-09-17T05:36:28.129905+0000","last_fullsized":"2024-09-17T05:36:28.129905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:06.019867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"31'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:31.529785+0000","last_change":"2024-09-17T05:27:26.663717+0000","last_active":"2024-09-17T05:36:31.529785+0000","last_peered":"2024-09-17T05:36:31.529785+0000","last_clean":"2024-09-17T05:36:31.529785+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:36:31.529785+0000","last_undegraded":"2024-09-17T05:36:31.529785+0000","last_fullsized":"2024-09-17T05:36:31.529785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:37.590340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"31'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:30.495721+0000","last_change":"2024-09-17T05:27:26.653019+0000","last_active":"2024-09-17T05:36:30.495721+0000","last_peered":"2024-09-17T05:36:30.495721+0000","last_clean":"2024-09-17T05:36:30.495721+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:36:30.495721+0000","last_undegraded":"2024-09-17T05:36:30.495721+0000","last_fullsized":"2024-09-17T05:36:30.495721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:02:47.529920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.140037+0000","last_change":"2024-09-17T05:36:37.610072+0000","last_active":"2024-09-17T05:36:38.140037+0000","last_peered":"2024-09-17T05:36:38.140037+0000","last_clean":"2024-09-17T05:36:38.140037+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:36:38.140037+0000","last_undegraded":"2024-09-17T05:36:38.140037+0000","last_fullsized":"2024-09-17T05:36:38.140037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:37.609959+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:37.609959+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.609959+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:03.953462+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"31'12","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:30.128475+0000","last_change":"2024-09-17T05:27:26.659787+0000","last_active":"2024-09-17T05:36:30.128475+0000","last_peered":"2024-09-17T05:36:30.128475+0000","last_clean":"2024-09-17T05:36:30.128475+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:36:30.128475+0000","last_undegraded":"2024-09-17T05:36:30.128475+0000","last_fullsized":"2024-09-17T05:36:30.128475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:39.379151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949435+0000","last_change":"2024-09-17T05:27:26.656888+0000","last_active":"2024-09-17T05:35:25.949435+0000","last_peered":"2024-09-17T05:35:25.949435+0000","last_clean":"2024-09-17T05:35:25.949435+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:35:25.949435+0000","last_undegraded":"2024-09-17T05:35:25.949435+0000","last_fullsized":"2024-09-17T05:35:25.949435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:55.685144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:33.063768+0000","last_change":"2024-09-17T05:27:26.661921+0000","last_active":"2024-09-17T05:36:33.063768+0000","last_peered":"2024-09-17T05:36:33.063768+0000","last_clean":"2024-09-17T05:36:33.063768+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:36:33.063768+0000","last_undegraded":"2024-09-17T05:36:33.063768+0000","last_fullsized":"2024-09-17T05:36:33.063768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:02.920431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:24.879099+0000","last_change":"2024-09-17T05:27:26.659406+0000","last_active":"2024-09-17T05:36:24.879099+0000","last_peered":"2024-09-17T05:36:24.879099+0000","last_clean":"2024-09-17T05:36:24.879099+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:36:24.879099+0000","last_undegraded":"2024-09-17T05:36:24.879099+0000","last_fullsized":"2024-09-17T05:36:24.879099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:15.230760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"31'12","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:29.939645+0000","last_change":"2024-09-17T05:27:26.663219+0000","last_active":"2024-09-17T05:36:29.939645+0000","last_peered":"2024-09-17T05:36:29.939645+0000","last_clean":"2024-09-17T05:36:29.939645+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:36:29.939645+0000","last_undegraded":"2024-09-17T05:36:29.939645+0000","last_fullsized":"2024-09-17T05:36:29.939645+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.357273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"31'13","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:36.759068+0000","last_change":"2024-09-17T05:27:26.656706+0000","last_active":"2024-09-17T05:36:36.759068+0000","last_peered":"2024-09-17T05:36:36.759068+0000","last_clean":"2024-09-17T05:36:36.759068+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:36:36.759068+0000","last_undegraded":"2024-09-17T05:36:36.759068+0000","last_fullsized":"2024-09-17T05:36:36.759068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:08.218030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"31'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.997285+0000","last_change":"2024-09-17T05:27:26.660277+0000","last_active":"2024-09-17T05:35:25.997285+0000","last_peered":"2024-09-17T05:35:25.997285+0000","last_clean":"2024-09-17T05:35:25.997285+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:35:25.997285+0000","last_undegraded":"2024-09-17T05:35:25.997285+0000","last_fullsized":"2024-09-17T05:35:25.997285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:48.984523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"31'10","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.946642+0000","last_change":"2024-09-17T05:27:26.660614+0000","last_active":"2024-09-17T05:36:37.946642+0000","last_peered":"2024-09-17T05:36:37.946642+0000","last_clean":"2024-09-17T05:36:37.946642+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:36:37.946642+0000","last_undegraded":"2024-09-17T05:36:37.946642+0000","last_fullsized":"2024-09-17T05:36:37.946642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"31'12","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.944560+0000","last_change":"2024-09-17T05:27:26.665557+0000","last_active":"2024-09-17T05:35:25.944560+0000","last_peered":"2024-09-17T05:35:25.944560+0000","last_clean":"2024-09-17T05:35:25.944560+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:35:25.944560+0000","last_undegraded":"2024-09-17T05:35:25.944560+0000","last_fullsized":"2024-09-17T05:35:25.944560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:34.011205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"31'13","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.991485+0000","last_change":"2024-09-17T05:27:26.658223+0000","last_active":"2024-09-17T05:36:37.991485+0000","last_peered":"2024-09-17T05:36:37.991485+0000","last_clean":"2024-09-17T05:36:37.991485+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:36:37.991485+0000","last_undegraded":"2024-09-17T05:36:37.991485+0000","last_fullsized":"2024-09-17T05:36:37.991485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"31'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:34.456668+0000","last_change":"2024-09-17T05:27:26.662362+0000","last_active":"2024-09-17T05:36:34.456668+0000","last_peered":"2024-09-17T05:36:34.456668+0000","last_clean":"2024-09-17T05:36:34.456668+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:36:34.456668+0000","last_undegraded":"2024-09-17T05:36:34.456668+0000","last_fullsized":"2024-09-17T05:36:34.456668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"31'4","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.949283+0000","last_change":"2024-09-17T05:27:26.663053+0000","last_active":"2024-09-17T05:35:25.949283+0000","last_peered":"2024-09-17T05:35:25.949283+0000","last_clean":"2024-09-17T05:35:25.949283+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:35:25.949283+0000","last_undegraded":"2024-09-17T05:35:25.949283+0000","last_fullsized":"2024-09-17T05:35:25.949283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:30.387408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"31'9","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.194264+0000","last_change":"2024-09-17T05:27:26.658408+0000","last_active":"2024-09-17T05:36:38.194264+0000","last_peered":"2024-09-17T05:36:38.194264+0000","last_clean":"2024-09-17T05:36:38.194264+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:36:38.194264+0000","last_undegraded":"2024-09-17T05:36:38.194264+0000","last_fullsized":"2024-09-17T05:36:38.194264+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:30.497470+0000","last_change":"2024-09-17T05:27:26.655060+0000","last_active":"2024-09-17T05:36:30.497470+0000","last_peered":"2024-09-17T05:36:30.497470+0000","last_clean":"2024-09-17T05:36:30.497470+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:36:30.497470+0000","last_undegraded":"2024-09-17T05:36:30.497470+0000","last_fullsized":"2024-09-17T05:36:30.497470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:09.881112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"31'15","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.232622+0000","last_change":"2024-09-17T05:27:26.663116+0000","last_active":"2024-09-17T05:36:38.232622+0000","last_peered":"2024-09-17T05:36:38.232622+0000","last_clean":"2024-09-17T05:36:38.232622+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:36:38.232622+0000","last_undegraded":"2024-09-17T05:36:38.232622+0000","last_fullsized":"2024-09-17T05:36:38.232622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"31'7","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:30.304589+0000","last_change":"2024-09-17T05:27:26.660999+0000","last_active":"2024-09-17T05:36:30.304589+0000","last_peered":"2024-09-17T05:36:30.304589+0000","last_clean":"2024-09-17T05:36:30.304589+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:36:30.304589+0000","last_undegraded":"2024-09-17T05:36:30.304589+0000","last_fullsized":"2024-09-17T05:36:30.304589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:16.719775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"31'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:25.578303+0000","last_change":"2024-09-17T05:27:26.657616+0000","last_active":"2024-09-17T05:36:25.578303+0000","last_peered":"2024-09-17T05:36:25.578303+0000","last_clean":"2024-09-17T05:36:25.578303+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:36:25.578303+0000","last_undegraded":"2024-09-17T05:36:25.578303+0000","last_fullsized":"2024-09-17T05:36:25.578303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:34.342255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"31'3","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:45.826185+0000","last_change":"2024-09-17T05:27:26.655212+0000","last_active":"2024-09-17T05:35:45.826185+0000","last_peered":"2024-09-17T05:35:45.826185+0000","last_clean":"2024-09-17T05:35:45.826185+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:35:45.826185+0000","last_undegraded":"2024-09-17T05:35:45.826185+0000","last_fullsized":"2024-09-17T05:35:45.826185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:39.928188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"28'4","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:35:25.947961+0000","last_change":"2024-09-17T05:28:07.483393+0000","last_active":"2024-09-17T05:35:25.947961+0000","last_peered":"2024-09-17T05:35:25.947961+0000","last_clean":"2024-09-17T05:35:25.947961+0000","last_became_active":"2024-09-17T05:28:07.483003+0000","last_became_peered":"2024-09-17T05:28:07.483003+0000","last_unstale":"2024-09-17T05:35:25.947961+0000","last_undegraded":"2024-09-17T05:35:25.947961+0000","last_fullsized":"2024-09-17T05:35:25.947961+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:50:27.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1625532542,"num_objects":564,"num_object_clones":0,"num_object_copies":2256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":525,"num_read_kb":0,"num_write":2621300,"num_write_kb":32721427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1661149184,"data_stored":3118583808,"data_compressed":162757842,"data_compressed_allocated":659550208,"data_compressed_original":2116984832,"omap_allocated":0,"internal_metadata":0},"log_size":177222,"ondisk_log_size":177222,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":82273271,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":349,"num_read_kb":78313,"num_write":2728,"num_write_kb":82426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34103296,"data_stored":164546542,"data_compressed":18858484,"data_compressed_allocated":21905408,"data_compressed_original":152502272,"omap_allocated":0,"internal_metadata":0},"log_size":2561,"ondisk_log_size":2561,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509552,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1771336,"kb_used_data":567672,"kb_used_omap":1,"kb_used_meta":1203646,"kb_avail":92600504,"statfs":{"total":96636764160,"available":94822916096,"internally_reserved":0,"allocated":581296128,"data_stored":1120852968,"data_compressed":60157267,"data_compressed_allocated":234401792,"data_compressed_original":774160384,"omap_allocated":1593,"internal_metadata":1232533959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509554,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1120104,"kb_used_data":283416,"kb_used_omap":1,"kb_used_meta":836670,"kb_avail":93251736,"statfs":{"total":96636764160,"available":95489777664,"internally_reserved":0,"allocated":290217984,"data_stored":550213011,"data_compressed":29633500,"data_compressed_allocated":116621312,"data_compressed_original":376815616,"omap_allocated":1595,"internal_metadata":856750533},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509554,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1454068,"kb_used_data":330724,"kb_used_omap":1,"kb_used_meta":1123326,"kb_avail":92917772,"statfs":{"total":96636764160,"available":95147798528,"internally_reserved":0,"allocated":338661376,"data_stored":658800631,"data_compressed":35911155,"data_compressed_allocated":135741440,"data_compressed_original":456081408,"omap_allocated":1595,"internal_metadata":1150286277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,7,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509555,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760480,"kb_used_data":29136,"kb_used_omap":1,"kb_used_meta":731326,"kb_avail":93611360,"statfs":{"total":96636764160,"available":95858032640,"internally_reserved":0,"allocated":29835264,"data_stored":80742056,"data_compressed":6219818,"data_compressed_allocated":15175680,"data_compressed_original":66289664,"omap_allocated":1595,"internal_metadata":748878277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542256,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1645408,"kb_used_data":540560,"kb_used_omap":1,"kb_used_meta":1104830,"kb_avail":92726432,"statfs":{"total":96636764160,"available":94951866368,"internally_reserved":0,"allocated":553533440,"data_stored":1073133210,"data_compressed":56380698,"data_compressed_allocated":225587200,"data_compressed_original":745402368,"omap_allocated":1590,"internal_metadata":1131346378},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542257,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1188056,"kb_used_data":104088,"kb_used_omap":1,"kb_used_meta":1082174,"kb_avail":93183784,"statfs":{"total":96636764160,"available":95420194816,"internally_reserved":0,"allocated":106586112,"data_stored":208710610,"data_compressed":11585429,"data_compressed_allocated":42536960,"data_compressed_original":144859136,"omap_allocated":1594,"internal_metadata":1108146630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,7,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542256,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1279688,"kb_used_data":388152,"kb_used_omap":1,"kb_used_meta":891518,"kb_avail":93092152,"statfs":{"total":96636764160,"available":95326363648,"internally_reserved":0,"allocated":397467648,"data_stored":755739537,"data_compressed":39888172,"data_compressed_allocated":158953472,"data_compressed_original":517451776,"omap_allocated":1594,"internal_metadata":912914886},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":967424,"kb_used_data":94064,"kb_used_omap":1,"kb_used_meta":873342,"kb_avail":93404416,"statfs":{"total":96636764160,"available":95646121984,"internally_reserved":0,"allocated":96321536,"data_stored":188357791,"data_compressed":11269218,"data_compressed_allocated":37728256,"data_compressed_original":129978368,"omap_allocated":1595,"internal_metadata":894302661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5029888,"data_stored":27648983,"data_compressed":3270621,"data_compressed_allocated":3801088,"data_compressed_original":26439680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":23452422,"data_compressed":2752708,"data_compressed_allocated":3211264,"data_compressed_original":22454272,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":12583618,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":37751063,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":12585152,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6701056,"data_stored":25261998,"data_compressed":2685898,"data_compressed_allocated":3158016,"data_compressed_original":21741568,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":12585217,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":12678089,"data_compressed":1499845,"data_compressed_allocated":1761280,"data_compressed_original":12292096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":178302976,"data_stored":351498240,"data_compressed":18738867,"data_compressed_allocated":73310208,"data_compressed_original":246505472,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264761344,"data_stored":493445120,"data_compressed":25747893,"data_compressed_allocated":105091072,"data_compressed_original":333774848,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":466751488,"data_stored":864854016,"data_compressed":44571935,"data_compressed_allocated":182747136,"data_compressed_original":580849664,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":267333632,"data_stored":501837824,"data_compressed":26293329,"data_compressed_allocated":106610688,"data_compressed_original":341114880,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":70987776,"data_stored":130904064,"data_compressed":7047063,"data_compressed_allocated":27934720,"data_compressed_original":87851008,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":3043328,"data_compressed":299273,"data_compressed_allocated":720896,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":411082752,"data_stored":773001216,"data_compressed":40059482,"data_compressed_allocated":163135488,"data_compressed_original":525053952,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:36:39.763 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.763 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.764 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.765 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.766 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.767 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T05:27:14.119421+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=14, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.768 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.769 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.770 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.771 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.772 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.772 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.772 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.772 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.772 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.773 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.773 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.773 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.773 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.774 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.774 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.774 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.775 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.775 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.775 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.775 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.776 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.777 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.777 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.777 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.777 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.777 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.778 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.778 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.778 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.778 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.779 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.779 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.779 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.779 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.779 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.779 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.780 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.780 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.780 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.780 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:36:39.781 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T05:36:59.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:36:59.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 -- 172.21.15.117:0/3413627533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4074fb0 msgr2=0x7f07c4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:59.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/3413627533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4074fb0 0x7f07c4072e10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f07b4005fd0 tx=0x7f07b402a8b0 comp rx=0 tx=0).stop 2024-09-17T05:36:59.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 -- 172.21.15.117:0/3413627533 shutdown_connections 2024-09-17T05:36:59.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/3413627533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c4074fb0 0x7f07c4072e10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:59.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 -- 172.21.15.117:0/3413627533 >> 172.21.15.117:0/3413627533 conn(0x7f07c406d190 msgr2=0x7f07c406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:36:59.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 -- 172.21.15.117:0/3413627533 shutdown_connections 2024-09-17T05:36:59.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.909+0000 7f07cb6dc640 1 -- 172.21.15.117:0/3413627533 wait complete. 2024-09-17T05:36:59.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.911+0000 7f07cb6dc640 1 Processor -- start 2024-09-17T05:36:59.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.911+0000 7f07cb6dc640 1 -- start start 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07cb6dc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07c40c4030 0x7f07c41ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07cb6dc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 0x7f07c4146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07cb6dc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f07c41476b0 0x7f07c4145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07cb6dc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f07c40c8a00 con 0x7f07c40c95f0 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07cb6dc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f07c4073350 con 0x7f07c40c4030 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07cb6dc640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f07c40c8b80 con 0x7f07c41476b0 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 0x7f07c4146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:59.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 0x7f07c4146810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55890/0 (socket says 172.21.15.117:55890) 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 -- 172.21.15.117:0/103918070 learned_addr learned my addr 172.21.15.117:0/103918070 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c9478640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07c40c4030 0x7f07c41ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c9c79640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f07c41476b0 0x7f07c4145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 -- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f07c41476b0 msgr2=0x7f07c4145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f07c41476b0 0x7f07c4145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 -- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07c40c4030 msgr2=0x7f07c41ee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07c40c4030 0x7f07c41ee5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.912+0000 7f07c8c77640 1 -- 172.21.15.117:0/103918070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c41eeb00 con 0x7f07c40c95f0 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.913+0000 7f07c8c77640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 0x7f07c4146810 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f07b800b3f0 tx=0x7f07b800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.913+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8015020 con 0x7f07c40c95f0 2024-09-17T05:36:59.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.913+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07b8011920 con 0x7f07c40c95f0 2024-09-17T05:36:59.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.913+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8011d70 con 0x7f07c40c95f0 2024-09-17T05:36:59.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.913+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c40c3330 con 0x7f07c40c95f0 2024-09-17T05:36:59.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.913+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c4144c50 con 0x7f07c40c95f0 2024-09-17T05:36:59.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.914+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07c41523e0 con 0x7f07c40c95f0 2024-09-17T05:36:59.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.915+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f07b8018020 con 0x7f07c40c95f0 2024-09-17T05:36:59.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.915+0000 7f07c1ffb640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f07b0078b90 0x7f07b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:36:59.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.915+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f07b809ad20 con 0x7f07c40c95f0 2024-09-17T05:36:59.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.915+0000 7f07c9478640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f07b0078b90 0x7f07b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:36:59.918 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.916+0000 7f07c9478640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f07b0078b90 0x7f07b007b0a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f07b4034000 tx=0x7f07b4004ab0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:36:59.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:36:59.920+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f07b801f030 con 0x7f07c40c95f0 2024-09-17T05:37:00.097 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.096+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07c414d810 con 0x7f07b0078b90 2024-09-17T05:37:00.104 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.102+0000 7f07c1ffb640 1 -- 172.21.15.117:0/103918070 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334733 (secure 0 0 0) 0x7f07c414d810 con 0x7f07b0078b90 2024-09-17T05:37:00.108 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:37:00.110 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:37:00.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f07b0078b90 msgr2=0x7f07b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f07b0078b90 0x7f07b007b0a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f07b4034000 tx=0x7f07b4004ab0 comp rx=0 tx=0).stop 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 msgr2=0x7f07c4146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 0x7f07c4146810 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f07b800b3f0 tx=0x7f07b800b8b0 comp rx=0 tx=0).stop 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 shutdown_connections 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f07b0078b90 0x7f07b007b0a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f07c41476b0 0x7f07c4145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07c40c95f0 0x7f07c4146810 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 --2- 172.21.15.117:0/103918070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07c40c4030 0x7f07c41ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:00.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 >> 172.21.15.117:0/103918070 conn(0x7f07c406d190 msgr2=0x7f07c4158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:00.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 shutdown_connections 2024-09-17T05:37:00.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:00.106+0000 7f07cb6dc640 1 -- 172.21.15.117:0/103918070 wait complete. 2024-09-17T05:37:00.118 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-17T05:36:59.538062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":82863639,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":940,"num_read_kb":78368,"num_write":2624701,"num_write_kb":32805482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15000,"num_omap_keys":30,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181338,"ondisk_log_size":181338,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7941304,"kb_used_data":36024,"kb_used_omap":12,"kb_used_meta":7905139,"kb_avail":747033416,"statfs":{"total":773094113280,"available":764962217984,"internally_reserved":0,"allocated":36888576,"data_stored":166879926,"data_compressed":18923458,"data_compressed_allocated":21995520,"data_compressed_original":153649152,"omap_allocated":12751,"internal_metadata":8094862897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003809"},"pg_stats":[{"pgid":"2.1f","version":"31'23","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.667369+0000","last_change":"2024-09-17T05:36:44.667369+0000","last_active":"2024-09-17T05:36:44.667369+0000","last_peered":"2024-09-17T05:36:44.667369+0000","last_clean":"2024-09-17T05:36:44.667369+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:36:44.667369+0000","last_undegraded":"2024-09-17T05:36:44.667369+0000","last_fullsized":"2024-09-17T05:36:44.667369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-09-17T05:36:44.667275+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-09-17T05:36:44.667275+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.667275+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:19:02.458753+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":4124,"num_write":15,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.211543+0000","last_change":"2024-09-17T05:36:43.211543+0000","last_active":"2024-09-17T05:36:43.211543+0000","last_peered":"2024-09-17T05:36:43.211543+0000","last_clean":"2024-09-17T05:36:43.211543+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:36:43.211543+0000","last_undegraded":"2024-09-17T05:36:43.211543+0000","last_fullsized":"2024-09-17T05:36:43.211543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-09-17T05:36:43.211450+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-09-17T05:36:43.211450+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.211450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:39:27.204598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"31'5864","reported_seq":5919,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.052321+0000","last_change":"2024-09-17T05:36:48.052321+0000","last_active":"2024-09-17T05:36:48.052321+0000","last_peered":"2024-09-17T05:36:48.052321+0000","last_clean":"2024-09-17T05:36:48.052321+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:36:48.052321+0000","last_undegraded":"2024-09-17T05:36:48.052321+0000","last_fullsized":"2024-09-17T05:36:48.052321+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5864","last_scrub_stamp":"2024-09-17T05:36:48.052212+0000","last_deep_scrub":"31'5864","last_deep_scrub_stamp":"2024-09-17T05:36:48.052212+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.052212+0000","objects_scrubbed":0,"log_size":5864,"log_dups_size":0,"ondisk_log_size":5864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:56.791747+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":86661,"num_write_kb":1098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"31'28","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.255358+0000","last_change":"2024-09-17T05:36:42.255358+0000","last_active":"2024-09-17T05:36:42.255358+0000","last_peered":"2024-09-17T05:36:42.255358+0000","last_clean":"2024-09-17T05:36:42.255358+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:36:42.255358+0000","last_undegraded":"2024-09-17T05:36:42.255358+0000","last_fullsized":"2024-09-17T05:36:42.255358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'28","last_scrub_stamp":"2024-09-17T05:36:42.255268+0000","last_deep_scrub":"31'28","last_deep_scrub_stamp":"2024-09-17T05:36:42.255268+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.255268+0000","objects_scrubbed":1,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:58.452012+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":19,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.181380+0000","last_change":"2024-09-17T05:36:47.181380+0000","last_active":"2024-09-17T05:36:47.181380+0000","last_peered":"2024-09-17T05:36:47.181380+0000","last_clean":"2024-09-17T05:36:47.181380+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:36:47.181380+0000","last_undegraded":"2024-09-17T05:36:47.181380+0000","last_fullsized":"2024-09-17T05:36:47.181380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:47.181292+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:47.181292+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.181292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:10.154989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"31'5772","reported_seq":5828,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.347203+0000","last_change":"2024-09-17T05:36:48.347203+0000","last_active":"2024-09-17T05:36:48.347203+0000","last_peered":"2024-09-17T05:36:48.347203+0000","last_clean":"2024-09-17T05:36:48.347203+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:36:48.347203+0000","last_undegraded":"2024-09-17T05:36:48.347203+0000","last_fullsized":"2024-09-17T05:36:48.347203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5772","last_scrub_stamp":"2024-09-17T05:36:48.347064+0000","last_deep_scrub":"31'5772","last_deep_scrub_stamp":"2024-09-17T05:36:48.347064+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.347064+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:57.310584+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":85101,"num_write_kb":1045061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"31'57","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.645973+0000","last_change":"2024-09-17T05:36:40.645973+0000","last_active":"2024-09-17T05:36:40.645973+0000","last_peered":"2024-09-17T05:36:40.645973+0000","last_clean":"2024-09-17T05:36:40.645973+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:36:40.645973+0000","last_undegraded":"2024-09-17T05:36:40.645973+0000","last_fullsized":"2024-09-17T05:36:40.645973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'57","last_scrub_stamp":"2024-09-17T05:36:40.645871+0000","last_deep_scrub":"31'57","last_deep_scrub_stamp":"2024-09-17T05:36:40.645871+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.645871+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:45.017498+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":8212,"num_write":40,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"31'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.704296+0000","last_change":"2024-09-17T05:36:35.704169+0000","last_active":"2024-09-17T05:36:35.704296+0000","last_peered":"2024-09-17T05:36:35.704296+0000","last_clean":"2024-09-17T05:36:35.704296+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:36:35.704296+0000","last_undegraded":"2024-09-17T05:36:35.704296+0000","last_fullsized":"2024-09-17T05:36:35.704296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'7","last_scrub_stamp":"2024-09-17T05:36:35.701723+0000","last_deep_scrub":"28'7","last_deep_scrub_stamp":"2024-09-17T05:36:35.701723+0000","last_clean_scrub_stamp":"2024-09-17T05:36:35.701723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:41.922299+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'5401","reported_seq":5458,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.336170+0000","last_change":"2024-09-17T05:36:50.336170+0000","last_active":"2024-09-17T05:36:50.336170+0000","last_peered":"2024-09-17T05:36:50.336170+0000","last_clean":"2024-09-17T05:36:50.336170+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:36:50.336170+0000","last_undegraded":"2024-09-17T05:36:50.336170+0000","last_fullsized":"2024-09-17T05:36:50.336170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5401","last_scrub_stamp":"2024-09-17T05:36:50.336047+0000","last_deep_scrub":"31'5401","last_deep_scrub_stamp":"2024-09-17T05:36:50.336047+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.336047+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:23:22.077821+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79473,"num_write_kb":982831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"31'205","reported_seq":254,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.674515+0000","last_change":"2024-09-17T05:36:35.674394+0000","last_active":"2024-09-17T05:36:35.674515+0000","last_peered":"2024-09-17T05:36:35.674515+0000","last_clean":"2024-09-17T05:36:35.674515+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:36:35.674515+0000","last_undegraded":"2024-09-17T05:36:35.674515+0000","last_fullsized":"2024-09-17T05:36:35.674515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'204","last_scrub_stamp":"2024-09-17T05:36:35.674263+0000","last_deep_scrub":"31'204","last_deep_scrub_stamp":"2024-09-17T05:36:35.674263+0000","last_clean_scrub_stamp":"2024-09-17T05:36:35.674263+0000","objects_scrubbed":2,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:04.566769+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2480742,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":83,"num_write":202,"num_write_kb":2537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.364939+0000","last_change":"2024-09-17T05:36:53.364939+0000","last_active":"2024-09-17T05:36:53.364939+0000","last_peered":"2024-09-17T05:36:53.364939+0000","last_clean":"2024-09-17T05:36:53.364939+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:36:53.364939+0000","last_undegraded":"2024-09-17T05:36:53.364939+0000","last_fullsized":"2024-09-17T05:36:53.364939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:53.364837+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:53.364837+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.364837+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:28:59.049700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"31'4634","reported_seq":4689,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.469730+0000","last_change":"2024-09-17T05:36:44.469730+0000","last_active":"2024-09-17T05:36:44.469730+0000","last_peered":"2024-09-17T05:36:44.469730+0000","last_clean":"2024-09-17T05:36:44.469730+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:36:44.469730+0000","last_undegraded":"2024-09-17T05:36:44.469730+0000","last_fullsized":"2024-09-17T05:36:44.469730+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4634","last_scrub_stamp":"2024-09-17T05:36:44.469613+0000","last_deep_scrub":"31'4634","last_deep_scrub_stamp":"2024-09-17T05:36:44.469613+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.469613+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:13.362120+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":68258,"num_write_kb":860103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:29.565935+0000","last_change":"2024-09-17T05:27:26.661670+0000","last_active":"2024-09-17T05:36:29.565935+0000","last_peered":"2024-09-17T05:36:29.565935+0000","last_clean":"2024-09-17T05:36:29.565935+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:36:29.565935+0000","last_undegraded":"2024-09-17T05:36:29.565935+0000","last_fullsized":"2024-09-17T05:36:29.565935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"31'133","reported_seq":184,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.514404+0000","last_change":"2024-09-17T05:36:44.514404+0000","last_active":"2024-09-17T05:36:44.514404+0000","last_peered":"2024-09-17T05:36:44.514404+0000","last_clean":"2024-09-17T05:36:44.514404+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:36:44.514404+0000","last_undegraded":"2024-09-17T05:36:44.514404+0000","last_fullsized":"2024-09-17T05:36:44.514404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'133","last_scrub_stamp":"2024-09-17T05:36:44.514300+0000","last_deep_scrub":"31'133","last_deep_scrub_stamp":"2024-09-17T05:36:44.514300+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.514300+0000","objects_scrubbed":3,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:14.968108+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4104,"num_write":119,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"31'5681","reported_seq":5723,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.266861+0000","last_change":"2024-09-17T05:27:30.686569+0000","last_active":"2024-09-17T05:36:38.266861+0000","last_peered":"2024-09-17T05:36:38.266861+0000","last_clean":"2024-09-17T05:36:38.266861+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:36:38.266861+0000","last_undegraded":"2024-09-17T05:36:38.266861+0000","last_fullsized":"2024-09-17T05:36:38.266861+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":83743,"num_write_kb":1023293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.027181+0000","last_change":"2024-09-17T05:36:41.027181+0000","last_active":"2024-09-17T05:36:41.027181+0000","last_peered":"2024-09-17T05:36:41.027181+0000","last_clean":"2024-09-17T05:36:41.027181+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:36:41.027181+0000","last_undegraded":"2024-09-17T05:36:41.027181+0000","last_fullsized":"2024-09-17T05:36:41.027181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:41.027074+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:41.027074+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.027074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:40.234211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"31'11","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.110113+0000","last_change":"2024-09-17T05:36:45.110113+0000","last_active":"2024-09-17T05:36:45.110113+0000","last_peered":"2024-09-17T05:36:45.110113+0000","last_clean":"2024-09-17T05:36:45.110113+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:36:45.110113+0000","last_undegraded":"2024-09-17T05:36:45.110113+0000","last_fullsized":"2024-09-17T05:36:45.110113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:45.110011+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:45.110011+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.110011+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:14.795400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"31'5558","reported_seq":5611,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.618305+0000","last_change":"2024-09-17T05:36:39.618305+0000","last_active":"2024-09-17T05:36:39.618305+0000","last_peered":"2024-09-17T05:36:39.618305+0000","last_clean":"2024-09-17T05:36:39.618305+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:36:39.618305+0000","last_undegraded":"2024-09-17T05:36:39.618305+0000","last_fullsized":"2024-09-17T05:36:39.618305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5558","last_scrub_stamp":"2024-09-17T05:36:39.618200+0000","last_deep_scrub":"31'5558","last_deep_scrub_stamp":"2024-09-17T05:36:39.618200+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.618200+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:48.581721+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":81978,"num_write_kb":1031804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.624406+0000","last_change":"2024-09-17T05:36:50.624406+0000","last_active":"2024-09-17T05:36:50.624406+0000","last_peered":"2024-09-17T05:36:50.624406+0000","last_clean":"2024-09-17T05:36:50.624406+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:36:50.624406+0000","last_undegraded":"2024-09-17T05:36:50.624406+0000","last_fullsized":"2024-09-17T05:36:50.624406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:50.624312+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:50.624312+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.624312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:11.176834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"31'9","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.661181+0000","last_change":"2024-09-17T05:36:52.661181+0000","last_active":"2024-09-17T05:36:52.661181+0000","last_peered":"2024-09-17T05:36:52.661181+0000","last_clean":"2024-09-17T05:36:52.661181+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:36:52.661181+0000","last_undegraded":"2024-09-17T05:36:52.661181+0000","last_fullsized":"2024-09-17T05:36:52.661181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:52.661090+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:52.661090+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.661090+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:50.537770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'6547","reported_seq":6595,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.567078+0000","last_change":"2024-09-17T05:36:51.567078+0000","last_active":"2024-09-17T05:36:51.567078+0000","last_peered":"2024-09-17T05:36:51.567078+0000","last_clean":"2024-09-17T05:36:51.567078+0000","last_became_active":"2024-09-17T05:28:07.483263+0000","last_became_peered":"2024-09-17T05:28:07.483263+0000","last_unstale":"2024-09-17T05:36:51.567078+0000","last_undegraded":"2024-09-17T05:36:51.567078+0000","last_fullsized":"2024-09-17T05:36:51.567078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6547","last_scrub_stamp":"2024-09-17T05:36:51.566959+0000","last_deep_scrub":"31'6547","last_deep_scrub_stamp":"2024-09-17T05:36:51.566959+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.566959+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:24:43.977342+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":96660,"num_write_kb":1196264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2,3],"acting":[0,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.407845+0000","last_change":"2024-09-17T05:36:41.407845+0000","last_active":"2024-09-17T05:36:41.407845+0000","last_peered":"2024-09-17T05:36:41.407845+0000","last_clean":"2024-09-17T05:36:41.407845+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:36:41.407845+0000","last_undegraded":"2024-09-17T05:36:41.407845+0000","last_fullsized":"2024-09-17T05:36:41.407845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:41.407755+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:41.407755+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.407755+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:34.958656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"31'17","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.014975+0000","last_change":"2024-09-17T05:36:34.672209+0000","last_active":"2024-09-17T05:36:38.014975+0000","last_peered":"2024-09-17T05:36:38.014975+0000","last_clean":"2024-09-17T05:36:38.014975+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:36:38.014975+0000","last_undegraded":"2024-09-17T05:36:38.014975+0000","last_fullsized":"2024-09-17T05:36:38.014975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:34.672070+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:34.672070+0000","last_clean_scrub_stamp":"2024-09-17T05:36:34.672070+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:12.516206+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":12,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"31'4858","reported_seq":4911,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.457817+0000","last_change":"2024-09-17T05:36:45.457817+0000","last_active":"2024-09-17T05:36:45.457817+0000","last_peered":"2024-09-17T05:36:45.457817+0000","last_clean":"2024-09-17T05:36:45.457817+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:36:45.457817+0000","last_undegraded":"2024-09-17T05:36:45.457817+0000","last_fullsized":"2024-09-17T05:36:45.457817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4858","last_scrub_stamp":"2024-09-17T05:36:45.457699+0000","last_deep_scrub":"31'4858","last_deep_scrub_stamp":"2024-09-17T05:36:45.457699+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.457699+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:18.190686+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":71425,"num_write_kb":889518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'19","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.585112+0000","last_change":"2024-09-17T05:36:42.585112+0000","last_active":"2024-09-17T05:36:42.585112+0000","last_peered":"2024-09-17T05:36:42.585112+0000","last_clean":"2024-09-17T05:36:42.585112+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:36:42.585112+0000","last_undegraded":"2024-09-17T05:36:42.585112+0000","last_fullsized":"2024-09-17T05:36:42.585112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:42.585017+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:42.585017+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.585017+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:15.318004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"31'10","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.113998+0000","last_change":"2024-09-17T05:36:37.501894+0000","last_active":"2024-09-17T05:36:38.113998+0000","last_peered":"2024-09-17T05:36:38.113998+0000","last_clean":"2024-09-17T05:36:38.113998+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:36:38.113998+0000","last_undegraded":"2024-09-17T05:36:38.113998+0000","last_fullsized":"2024-09-17T05:36:38.113998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:37.501768+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:37.501768+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.501768+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:58.829230+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"31'5547","reported_seq":5604,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.416286+0000","last_change":"2024-09-17T05:36:46.416286+0000","last_active":"2024-09-17T05:36:46.416286+0000","last_peered":"2024-09-17T05:36:46.416286+0000","last_clean":"2024-09-17T05:36:46.416286+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:36:46.416286+0000","last_undegraded":"2024-09-17T05:36:46.416286+0000","last_fullsized":"2024-09-17T05:36:46.416286+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5547","last_scrub_stamp":"2024-09-17T05:36:46.416107+0000","last_deep_scrub":"31'5547","last_deep_scrub_stamp":"2024-09-17T05:36:46.416107+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.416107+0000","objects_scrubbed":0,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:12.759610+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":81766,"num_write_kb":1039133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.625891+0000","last_change":"2024-09-17T05:36:38.625891+0000","last_active":"2024-09-17T05:36:38.625891+0000","last_peered":"2024-09-17T05:36:38.625891+0000","last_clean":"2024-09-17T05:36:38.625891+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:36:38.625891+0000","last_undegraded":"2024-09-17T05:36:38.625891+0000","last_fullsized":"2024-09-17T05:36:38.625891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:38.625752+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:38.625752+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.625752+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:05.595811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"31'225","reported_seq":277,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.602460+0000","last_change":"2024-09-17T05:36:40.602460+0000","last_active":"2024-09-17T05:36:40.602460+0000","last_peered":"2024-09-17T05:36:40.602460+0000","last_clean":"2024-09-17T05:36:40.602460+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:36:40.602460+0000","last_undegraded":"2024-09-17T05:36:40.602460+0000","last_fullsized":"2024-09-17T05:36:40.602460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'225","last_scrub_stamp":"2024-09-17T05:36:40.602363+0000","last_deep_scrub":"31'225","last_deep_scrub_stamp":"2024-09-17T05:36:40.602363+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.602363+0000","objects_scrubbed":2,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.287752+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":21,"num_write":446,"num_write_kb":181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"31'6023","reported_seq":6082,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.684015+0000","last_change":"2024-09-17T05:36:51.684015+0000","last_active":"2024-09-17T05:36:51.684015+0000","last_peered":"2024-09-17T05:36:51.684015+0000","last_clean":"2024-09-17T05:36:51.684015+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:36:51.684015+0000","last_undegraded":"2024-09-17T05:36:51.684015+0000","last_fullsized":"2024-09-17T05:36:51.684015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6023","last_scrub_stamp":"2024-09-17T05:36:51.683904+0000","last_deep_scrub":"31'6023","last_deep_scrub_stamp":"2024-09-17T05:36:51.683904+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.683904+0000","objects_scrubbed":0,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:05:39.444905+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":88895,"num_write_kb":1123429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.032926+0000","last_change":"2024-09-17T05:36:41.032926+0000","last_active":"2024-09-17T05:36:41.032926+0000","last_peered":"2024-09-17T05:36:41.032926+0000","last_clean":"2024-09-17T05:36:41.032926+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:36:41.032926+0000","last_undegraded":"2024-09-17T05:36:41.032926+0000","last_fullsized":"2024-09-17T05:36:41.032926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:41.032832+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:41.032832+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.032832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:57.520423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"31'6","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.448891+0000","last_change":"2024-09-17T05:36:51.448891+0000","last_active":"2024-09-17T05:36:51.448891+0000","last_peered":"2024-09-17T05:36:51.448891+0000","last_clean":"2024-09-17T05:36:51.448891+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:36:51.448891+0000","last_undegraded":"2024-09-17T05:36:51.448891+0000","last_fullsized":"2024-09-17T05:36:51.448891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-09-17T05:36:51.448792+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-09-17T05:36:51.448792+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.448792+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:32.191573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'5078","reported_seq":5128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.279852+0000","last_change":"2024-09-17T05:36:50.279852+0000","last_active":"2024-09-17T05:36:50.279852+0000","last_peered":"2024-09-17T05:36:50.279852+0000","last_clean":"2024-09-17T05:36:50.279852+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:36:50.279852+0000","last_undegraded":"2024-09-17T05:36:50.279852+0000","last_fullsized":"2024-09-17T05:36:50.279852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5078","last_scrub_stamp":"2024-09-17T05:36:50.279730+0000","last_deep_scrub":"31'5078","last_deep_scrub_stamp":"2024-09-17T05:36:50.279730+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.279730+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:59.243251+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":75020,"num_write_kb":939196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.323231+0000","last_change":"2024-09-17T05:36:38.323231+0000","last_active":"2024-09-17T05:36:38.323231+0000","last_peered":"2024-09-17T05:36:38.323231+0000","last_clean":"2024-09-17T05:36:38.323231+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:36:38.323231+0000","last_undegraded":"2024-09-17T05:36:38.323231+0000","last_fullsized":"2024-09-17T05:36:38.323231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:38.323156+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:38.323156+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.323156+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:31.500391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"31'5","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.220388+0000","last_change":"2024-09-17T05:36:44.220388+0000","last_active":"2024-09-17T05:36:44.220388+0000","last_peered":"2024-09-17T05:36:44.220388+0000","last_clean":"2024-09-17T05:36:44.220388+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:36:44.220388+0000","last_undegraded":"2024-09-17T05:36:44.220388+0000","last_fullsized":"2024-09-17T05:36:44.220388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-09-17T05:36:44.220283+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-09-17T05:36:44.220283+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.220283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:09.189959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"31'6224","reported_seq":6280,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.336045+0000","last_change":"2024-09-17T05:36:52.336045+0000","last_active":"2024-09-17T05:36:52.336045+0000","last_peered":"2024-09-17T05:36:52.336045+0000","last_clean":"2024-09-17T05:36:52.336045+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:36:52.336045+0000","last_undegraded":"2024-09-17T05:36:52.336045+0000","last_fullsized":"2024-09-17T05:36:52.336045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6224","last_scrub_stamp":"2024-09-17T05:36:52.335891+0000","last_deep_scrub":"31'6224","last_deep_scrub_stamp":"2024-09-17T05:36:52.335891+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.335891+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:40:59.226496+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":92055,"num_write_kb":1145505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.685814+0000","last_change":"2024-09-17T05:36:47.685814+0000","last_active":"2024-09-17T05:36:47.685814+0000","last_peered":"2024-09-17T05:36:47.685814+0000","last_clean":"2024-09-17T05:36:47.685814+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:36:47.685814+0000","last_undegraded":"2024-09-17T05:36:47.685814+0000","last_fullsized":"2024-09-17T05:36:47.685814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:47.685701+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:47.685701+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.685701+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:08.285051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"31'45","reported_seq":86,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.110024+0000","last_change":"2024-09-17T05:27:25.653181+0000","last_active":"2024-09-17T05:36:38.110024+0000","last_peered":"2024-09-17T05:36:38.110024+0000","last_clean":"2024-09-17T05:36:38.110024+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:36:38.110024+0000","last_undegraded":"2024-09-17T05:36:38.110024+0000","last_fullsized":"2024-09-17T05:36:38.110024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":35,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'5360","reported_seq":5418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.256733+0000","last_change":"2024-09-17T05:36:49.256733+0000","last_active":"2024-09-17T05:36:49.256733+0000","last_peered":"2024-09-17T05:36:49.256733+0000","last_clean":"2024-09-17T05:36:49.256733+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:36:49.256733+0000","last_undegraded":"2024-09-17T05:36:49.256733+0000","last_fullsized":"2024-09-17T05:36:49.256733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5360","last_scrub_stamp":"2024-09-17T05:36:49.256614+0000","last_deep_scrub":"31'5360","last_deep_scrub_stamp":"2024-09-17T05:36:49.256614+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.256614+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:19:07.048092+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79033,"num_write_kb":985615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.248530+0000","last_change":"2024-09-17T05:36:41.248530+0000","last_active":"2024-09-17T05:36:41.248530+0000","last_peered":"2024-09-17T05:36:41.248530+0000","last_clean":"2024-09-17T05:36:41.248530+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:36:41.248530+0000","last_undegraded":"2024-09-17T05:36:41.248530+0000","last_fullsized":"2024-09-17T05:36:41.248530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:41.248425+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:41.248425+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.248425+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:20.776332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"31'33","reported_seq":102,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.066242+0000","last_change":"2024-09-17T05:36:40.066242+0000","last_active":"2024-09-17T05:36:40.066242+0000","last_peered":"2024-09-17T05:36:40.066242+0000","last_clean":"2024-09-17T05:36:40.066242+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:36:40.066242+0000","last_undegraded":"2024-09-17T05:36:40.066242+0000","last_fullsized":"2024-09-17T05:36:40.066242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'33","last_scrub_stamp":"2024-09-17T05:36:40.066094+0000","last_deep_scrub":"31'33","last_deep_scrub_stamp":"2024-09-17T05:36:40.066094+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.066094+0000","objects_scrubbed":3,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:02.733563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94359,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":116,"num_write":27,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'5929","reported_seq":5988,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.307148+0000","last_change":"2024-09-17T05:36:53.307148+0000","last_active":"2024-09-17T05:36:53.307148+0000","last_peered":"2024-09-17T05:36:53.307148+0000","last_clean":"2024-09-17T05:36:53.307148+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:36:53.307148+0000","last_undegraded":"2024-09-17T05:36:53.307148+0000","last_fullsized":"2024-09-17T05:36:53.307148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5929","last_scrub_stamp":"2024-09-17T05:36:53.307008+0000","last_deep_scrub":"31'5929","last_deep_scrub_stamp":"2024-09-17T05:36:53.307008+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.307008+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:20.214495+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":87395,"num_write_kb":1093347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"31'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.078516+0000","last_change":"2024-09-17T05:36:42.078516+0000","last_active":"2024-09-17T05:36:42.078516+0000","last_peered":"2024-09-17T05:36:42.078516+0000","last_clean":"2024-09-17T05:36:42.078516+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:36:42.078516+0000","last_undegraded":"2024-09-17T05:36:42.078516+0000","last_fullsized":"2024-09-17T05:36:42.078516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:42.078430+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:42.078430+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.078430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:48.066042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"31'98","reported_seq":151,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.485406+0000","last_change":"2024-09-17T05:36:46.485406+0000","last_active":"2024-09-17T05:36:46.485406+0000","last_peered":"2024-09-17T05:36:46.485406+0000","last_clean":"2024-09-17T05:36:46.485406+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:36:46.485406+0000","last_undegraded":"2024-09-17T05:36:46.485406+0000","last_fullsized":"2024-09-17T05:36:46.485406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'98","last_scrub_stamp":"2024-09-17T05:36:46.485320+0000","last_deep_scrub":"31'98","last_deep_scrub_stamp":"2024-09-17T05:36:46.485320+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.485320+0000","objects_scrubbed":3,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:27.174887+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8204,"num_write":88,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"31'4825","reported_seq":4875,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:55.355859+0000","last_change":"2024-09-17T05:36:55.355859+0000","last_active":"2024-09-17T05:36:55.355859+0000","last_peered":"2024-09-17T05:36:55.355859+0000","last_clean":"2024-09-17T05:36:55.355859+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:36:55.355859+0000","last_undegraded":"2024-09-17T05:36:55.355859+0000","last_fullsized":"2024-09-17T05:36:55.355859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4825","last_scrub_stamp":"2024-09-17T05:36:55.355734+0000","last_deep_scrub":"31'4825","last_deep_scrub_stamp":"2024-09-17T05:36:55.355734+0000","last_clean_scrub_stamp":"2024-09-17T05:36:55.355734+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:58.879206+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":71019,"num_write_kb":889582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"31'8","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.701641+0000","last_change":"2024-09-17T05:36:45.701641+0000","last_active":"2024-09-17T05:36:45.701641+0000","last_peered":"2024-09-17T05:36:45.701641+0000","last_clean":"2024-09-17T05:36:45.701641+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:36:45.701641+0000","last_undegraded":"2024-09-17T05:36:45.701641+0000","last_fullsized":"2024-09-17T05:36:45.701641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:45.701541+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:45.701541+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.701541+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:54.665061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'9","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.432601+0000","last_change":"2024-09-17T05:36:42.432601+0000","last_active":"2024-09-17T05:36:42.432601+0000","last_peered":"2024-09-17T05:36:42.432601+0000","last_clean":"2024-09-17T05:36:42.432601+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:36:42.432601+0000","last_undegraded":"2024-09-17T05:36:42.432601+0000","last_fullsized":"2024-09-17T05:36:42.432601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:42.432508+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:42.432508+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.432508+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:00.017778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"31'74","reported_seq":145,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:36.709521+0000","last_change":"2024-09-17T05:36:36.709521+0000","last_active":"2024-09-17T05:36:36.709521+0000","last_peered":"2024-09-17T05:36:36.709521+0000","last_clean":"2024-09-17T05:36:36.709521+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:36:36.709521+0000","last_undegraded":"2024-09-17T05:36:36.709521+0000","last_fullsized":"2024-09-17T05:36:36.709521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'74","last_scrub_stamp":"2024-09-17T05:36:36.703771+0000","last_deep_scrub":"31'74","last_deep_scrub_stamp":"2024-09-17T05:36:36.703771+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.703771+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:32.886636+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'5418","reported_seq":5467,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.172068+0000","last_change":"2024-09-17T05:27:30.673905+0000","last_active":"2024-09-17T05:36:38.172068+0000","last_peered":"2024-09-17T05:36:38.172068+0000","last_clean":"2024-09-17T05:36:38.172068+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:36:38.172068+0000","last_undegraded":"2024-09-17T05:36:38.172068+0000","last_fullsized":"2024-09-17T05:36:38.172068+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":80049,"num_write_kb":1015697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.403680+0000","last_change":"2024-09-17T05:36:52.403680+0000","last_active":"2024-09-17T05:36:52.403680+0000","last_peered":"2024-09-17T05:36:52.403680+0000","last_clean":"2024-09-17T05:36:52.403680+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:36:52.403680+0000","last_undegraded":"2024-09-17T05:36:52.403680+0000","last_fullsized":"2024-09-17T05:36:52.403680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:52.403589+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:52.403589+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.403589+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:05.144325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.136774+0000","last_change":"2024-09-17T05:36:44.136774+0000","last_active":"2024-09-17T05:36:44.136774+0000","last_peered":"2024-09-17T05:36:44.136774+0000","last_clean":"2024-09-17T05:36:44.136774+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:36:44.136774+0000","last_undegraded":"2024-09-17T05:36:44.136774+0000","last_fullsized":"2024-09-17T05:36:44.136774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:44.136684+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:44.136684+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.136684+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:19.125250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"31'5576","reported_seq":5632,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.433064+0000","last_change":"2024-09-17T05:36:43.433064+0000","last_active":"2024-09-17T05:36:43.433064+0000","last_peered":"2024-09-17T05:36:43.433064+0000","last_clean":"2024-09-17T05:36:43.433064+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:36:43.433064+0000","last_undegraded":"2024-09-17T05:36:43.433064+0000","last_fullsized":"2024-09-17T05:36:43.433064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5576","last_scrub_stamp":"2024-09-17T05:36:43.432953+0000","last_deep_scrub":"31'5576","last_deep_scrub_stamp":"2024-09-17T05:36:43.432953+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.432953+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:21.277089+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82316,"num_write_kb":1025442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'14","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.139906+0000","last_change":"2024-09-17T05:36:36.597250+0000","last_active":"2024-09-17T05:36:38.139906+0000","last_peered":"2024-09-17T05:36:38.139906+0000","last_clean":"2024-09-17T05:36:38.139906+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:36:38.139906+0000","last_undegraded":"2024-09-17T05:36:38.139906+0000","last_fullsized":"2024-09-17T05:36:38.139906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:36.597064+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:36.597064+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.597064+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:09.330050+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"31'23","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.253255+0000","last_change":"2024-09-17T05:36:40.253255+0000","last_active":"2024-09-17T05:36:40.253255+0000","last_peered":"2024-09-17T05:36:40.253255+0000","last_clean":"2024-09-17T05:36:40.253255+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:36:40.253255+0000","last_undegraded":"2024-09-17T05:36:40.253255+0000","last_fullsized":"2024-09-17T05:36:40.253255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-09-17T05:36:40.253130+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-09-17T05:36:40.253130+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.253130+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:45.419407+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":19,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"31'5086","reported_seq":5143,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:56.376470+0000","last_change":"2024-09-17T05:36:56.376470+0000","last_active":"2024-09-17T05:36:56.376470+0000","last_peered":"2024-09-17T05:36:56.376470+0000","last_clean":"2024-09-17T05:36:56.376470+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:36:56.376470+0000","last_undegraded":"2024-09-17T05:36:56.376470+0000","last_fullsized":"2024-09-17T05:36:56.376470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5086","last_scrub_stamp":"2024-09-17T05:36:56.376313+0000","last_deep_scrub":"31'5086","last_deep_scrub_stamp":"2024-09-17T05:36:56.376313+0000","last_clean_scrub_stamp":"2024-09-17T05:36:56.376313+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:31.672873+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":74966,"num_write_kb":933970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.548955+0000","last_change":"2024-09-17T05:36:50.548955+0000","last_active":"2024-09-17T05:36:50.548955+0000","last_peered":"2024-09-17T05:36:50.548955+0000","last_clean":"2024-09-17T05:36:50.548955+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:36:50.548955+0000","last_undegraded":"2024-09-17T05:36:50.548955+0000","last_fullsized":"2024-09-17T05:36:50.548955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:50.548871+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:50.548871+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.548871+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:28.306780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"31'32","reported_seq":83,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.618965+0000","last_change":"2024-09-17T05:36:41.618965+0000","last_active":"2024-09-17T05:36:41.618965+0000","last_peered":"2024-09-17T05:36:41.618965+0000","last_clean":"2024-09-17T05:36:41.618965+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:36:41.618965+0000","last_undegraded":"2024-09-17T05:36:41.618965+0000","last_fullsized":"2024-09-17T05:36:41.618965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'32","last_scrub_stamp":"2024-09-17T05:36:41.618873+0000","last_deep_scrub":"31'32","last_deep_scrub_stamp":"2024-09-17T05:36:41.618873+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.618873+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:10.511379+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":26,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"31'5181","reported_seq":5234,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.491487+0000","last_change":"2024-09-17T05:36:50.491487+0000","last_active":"2024-09-17T05:36:50.491487+0000","last_peered":"2024-09-17T05:36:50.491487+0000","last_clean":"2024-09-17T05:36:50.491487+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:36:50.491487+0000","last_undegraded":"2024-09-17T05:36:50.491487+0000","last_fullsized":"2024-09-17T05:36:50.491487+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5181","last_scrub_stamp":"2024-09-17T05:36:50.491356+0000","last_deep_scrub":"31'5181","last_deep_scrub_stamp":"2024-09-17T05:36:50.491356+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.491356+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:04.357783+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76474,"num_write_kb":955028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"31'7","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.927749+0000","last_change":"2024-09-17T05:36:37.310355+0000","last_active":"2024-09-17T05:36:37.927749+0000","last_peered":"2024-09-17T05:36:37.927749+0000","last_clean":"2024-09-17T05:36:37.927749+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:36:37.927749+0000","last_undegraded":"2024-09-17T05:36:37.927749+0000","last_fullsized":"2024-09-17T05:36:37.927749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-09-17T05:36:37.309821+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-09-17T05:36:37.309821+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.309821+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:57.303315+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"31'13","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.598972+0000","last_change":"2024-09-17T05:36:49.598972+0000","last_active":"2024-09-17T05:36:49.598972+0000","last_peered":"2024-09-17T05:36:49.598972+0000","last_clean":"2024-09-17T05:36:49.598972+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:36:49.598972+0000","last_undegraded":"2024-09-17T05:36:49.598972+0000","last_fullsized":"2024-09-17T05:36:49.598972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:49.598830+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:49.598830+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.598830+0000","objects_scrubbed":2,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:32.336296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"31'5199","reported_seq":5252,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.765591+0000","last_change":"2024-09-17T05:36:40.765591+0000","last_active":"2024-09-17T05:36:40.765591+0000","last_peered":"2024-09-17T05:36:40.765591+0000","last_clean":"2024-09-17T05:36:40.765591+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:36:40.765591+0000","last_undegraded":"2024-09-17T05:36:40.765591+0000","last_fullsized":"2024-09-17T05:36:40.765591+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5199","last_scrub_stamp":"2024-09-17T05:36:40.765481+0000","last_deep_scrub":"31'5199","last_deep_scrub_stamp":"2024-09-17T05:36:40.765481+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.765481+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:03.432950+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76687,"num_write_kb":970760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.453257+0000","last_change":"2024-09-17T05:36:43.453147+0000","last_active":"2024-09-17T05:36:43.453257+0000","last_peered":"2024-09-17T05:36:43.453257+0000","last_clean":"2024-09-17T05:36:43.453257+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:36:43.453257+0000","last_undegraded":"2024-09-17T05:36:43.453257+0000","last_fullsized":"2024-09-17T05:36:43.453257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:43.453017+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:43.453017+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.453017+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:42:46.999337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"31'31","reported_seq":94,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.662904+0000","last_change":"2024-09-17T05:36:49.662904+0000","last_active":"2024-09-17T05:36:49.662904+0000","last_peered":"2024-09-17T05:36:49.662904+0000","last_clean":"2024-09-17T05:36:49.662904+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:36:49.662904+0000","last_undegraded":"2024-09-17T05:36:49.662904+0000","last_fullsized":"2024-09-17T05:36:49.662904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'31","last_scrub_stamp":"2024-09-17T05:36:49.662789+0000","last_deep_scrub":"31'31","last_deep_scrub_stamp":"2024-09-17T05:36:49.662789+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.662789+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:14.581798+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":40,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"31'5828","reported_seq":5879,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.068705+0000","last_change":"2024-09-17T05:27:30.689121+0000","last_active":"2024-09-17T05:36:38.068705+0000","last_peered":"2024-09-17T05:36:38.068705+0000","last_clean":"2024-09-17T05:36:38.068705+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:36:38.068705+0000","last_undegraded":"2024-09-17T05:36:38.068705+0000","last_fullsized":"2024-09-17T05:36:38.068705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:34.618293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":85918,"num_write_kb":1086559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.173409+0000","last_change":"2024-09-17T05:36:37.673724+0000","last_active":"2024-09-17T05:36:38.173409+0000","last_peered":"2024-09-17T05:36:38.173409+0000","last_clean":"2024-09-17T05:36:38.173409+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:36:38.173409+0000","last_undegraded":"2024-09-17T05:36:38.173409+0000","last_fullsized":"2024-09-17T05:36:38.173409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:37.673615+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:37.673615+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.673615+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:25.551966+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"31'19","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.712843+0000","last_change":"2024-09-17T05:36:43.712843+0000","last_active":"2024-09-17T05:36:43.712843+0000","last_peered":"2024-09-17T05:36:43.712843+0000","last_clean":"2024-09-17T05:36:43.712843+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:36:43.712843+0000","last_undegraded":"2024-09-17T05:36:43.712843+0000","last_fullsized":"2024-09-17T05:36:43.712843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:43.712680+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:43.712680+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.712680+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:08.631689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":14,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"31'5926","reported_seq":5974,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.224388+0000","last_change":"2024-09-17T05:27:30.686095+0000","last_active":"2024-09-17T05:36:38.224388+0000","last_peered":"2024-09-17T05:36:38.224388+0000","last_clean":"2024-09-17T05:36:38.224388+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:36:38.224388+0000","last_undegraded":"2024-09-17T05:36:38.224388+0000","last_fullsized":"2024-09-17T05:36:38.224388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:14.984937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87434,"num_write_kb":1070021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.496662+0000","last_change":"2024-09-17T05:36:45.496592+0000","last_active":"2024-09-17T05:36:45.496662+0000","last_peered":"2024-09-17T05:36:45.496662+0000","last_clean":"2024-09-17T05:36:45.496662+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:36:45.496662+0000","last_undegraded":"2024-09-17T05:36:45.496662+0000","last_fullsized":"2024-09-17T05:36:45.496662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:45.496501+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:45.496501+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.496501+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:59.345367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"31'10","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.243578+0000","last_change":"2024-09-17T05:27:25.654652+0000","last_active":"2024-09-17T05:36:38.243578+0000","last_peered":"2024-09-17T05:36:38.243578+0000","last_clean":"2024-09-17T05:36:38.243578+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:36:38.243578+0000","last_undegraded":"2024-09-17T05:36:38.243578+0000","last_fullsized":"2024-09-17T05:36:38.243578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'5945","reported_seq":6002,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.314752+0000","last_change":"2024-09-17T05:36:51.314677+0000","last_active":"2024-09-17T05:36:51.314752+0000","last_peered":"2024-09-17T05:36:51.314752+0000","last_clean":"2024-09-17T05:36:51.314752+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:36:51.314752+0000","last_undegraded":"2024-09-17T05:36:51.314752+0000","last_fullsized":"2024-09-17T05:36:51.314752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5945","last_scrub_stamp":"2024-09-17T05:36:51.314565+0000","last_deep_scrub":"31'5945","last_deep_scrub_stamp":"2024-09-17T05:36:51.314565+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.314565+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:55.686191+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87608,"num_write_kb":1085839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.018029+0000","last_change":"2024-09-17T05:36:40.017952+0000","last_active":"2024-09-17T05:36:40.018029+0000","last_peered":"2024-09-17T05:36:40.018029+0000","last_clean":"2024-09-17T05:36:40.018029+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:36:40.018029+0000","last_undegraded":"2024-09-17T05:36:40.018029+0000","last_fullsized":"2024-09-17T05:36:40.018029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:40.017855+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:40.017855+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.017855+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:20.964313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"31'98","reported_seq":154,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.067437+0000","last_change":"2024-09-17T05:36:47.067348+0000","last_active":"2024-09-17T05:36:47.067437+0000","last_peered":"2024-09-17T05:36:47.067437+0000","last_clean":"2024-09-17T05:36:47.067437+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:36:47.067437+0000","last_undegraded":"2024-09-17T05:36:47.067437+0000","last_fullsized":"2024-09-17T05:36:47.067437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'98","last_scrub_stamp":"2024-09-17T05:36:47.067218+0000","last_deep_scrub":"31'98","last_deep_scrub_stamp":"2024-09-17T05:36:47.067218+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.067218+0000","objects_scrubbed":6,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:07.060712+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":8218,"num_write":88,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"31'5226","reported_seq":5272,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.687237+0000","last_change":"2024-09-17T05:27:30.678669+0000","last_active":"2024-09-17T05:36:37.687237+0000","last_peered":"2024-09-17T05:36:37.687237+0000","last_clean":"2024-09-17T05:36:37.687237+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:36:37.687237+0000","last_undegraded":"2024-09-17T05:36:37.687237+0000","last_fullsized":"2024-09-17T05:36:37.687237+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:29.653148+0000","last_clean_scrub_stamp":"2024-09-17T05:27:29.653148+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":76989,"num_write_kb":960359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"31'11","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.661192+0000","last_change":"2024-09-17T05:36:41.661192+0000","last_active":"2024-09-17T05:36:41.661192+0000","last_peered":"2024-09-17T05:36:41.661192+0000","last_clean":"2024-09-17T05:36:41.661192+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:36:41.661192+0000","last_undegraded":"2024-09-17T05:36:41.661192+0000","last_fullsized":"2024-09-17T05:36:41.661192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:41.661044+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:41.661044+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.661044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:40:48.551650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"31'221","reported_seq":275,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.103803+0000","last_change":"2024-09-17T05:36:36.402251+0000","last_active":"2024-09-17T05:36:38.103803+0000","last_peered":"2024-09-17T05:36:38.103803+0000","last_clean":"2024-09-17T05:36:38.103803+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:36:38.103803+0000","last_undegraded":"2024-09-17T05:36:38.103803+0000","last_fullsized":"2024-09-17T05:36:38.103803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'218","last_scrub_stamp":"2024-09-17T05:36:36.402105+0000","last_deep_scrub":"31'218","last_deep_scrub_stamp":"2024-09-17T05:36:36.402105+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.402105+0000","objects_scrubbed":5,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:36.078027+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":213,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"31'5421","reported_seq":5479,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.718298+0000","last_change":"2024-09-17T05:36:46.718223+0000","last_active":"2024-09-17T05:36:46.718298+0000","last_peered":"2024-09-17T05:36:46.718298+0000","last_clean":"2024-09-17T05:36:46.718298+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:36:46.718298+0000","last_undegraded":"2024-09-17T05:36:46.718298+0000","last_fullsized":"2024-09-17T05:36:46.718298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5421","last_scrub_stamp":"2024-09-17T05:36:46.718099+0000","last_deep_scrub":"31'5421","last_deep_scrub_stamp":"2024-09-17T05:36:46.718099+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.718099+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:51.089726+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79593,"num_write_kb":1008039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.442060+0000","last_change":"2024-09-17T05:36:40.441990+0000","last_active":"2024-09-17T05:36:40.442060+0000","last_peered":"2024-09-17T05:36:40.442060+0000","last_clean":"2024-09-17T05:36:40.442060+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:36:40.442060+0000","last_undegraded":"2024-09-17T05:36:40.442060+0000","last_fullsized":"2024-09-17T05:36:40.442060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:40.441903+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:40.441903+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.441903+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:24:32.852286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"31'15","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.004273+0000","last_change":"2024-09-17T05:36:43.004273+0000","last_active":"2024-09-17T05:36:43.004273+0000","last_peered":"2024-09-17T05:36:43.004273+0000","last_clean":"2024-09-17T05:36:43.004273+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:36:43.004273+0000","last_undegraded":"2024-09-17T05:36:43.004273+0000","last_fullsized":"2024-09-17T05:36:43.004273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:43.004159+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:43.004159+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.004159+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:36.614676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"31'5687","reported_seq":5741,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.370300+0000","last_change":"2024-09-17T05:36:49.370233+0000","last_active":"2024-09-17T05:36:49.370300+0000","last_peered":"2024-09-17T05:36:49.370300+0000","last_clean":"2024-09-17T05:36:49.370300+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:36:49.370300+0000","last_undegraded":"2024-09-17T05:36:49.370300+0000","last_fullsized":"2024-09-17T05:36:49.370300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5687","last_scrub_stamp":"2024-09-17T05:36:49.370130+0000","last_deep_scrub":"31'5687","last_deep_scrub_stamp":"2024-09-17T05:36:49.370130+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.370130+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:01.118430+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":0,"num_write":83481,"num_write_kb":1022097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'16","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.193442+0000","last_change":"2024-09-17T05:36:46.193442+0000","last_active":"2024-09-17T05:36:46.193442+0000","last_peered":"2024-09-17T05:36:46.193442+0000","last_clean":"2024-09-17T05:36:46.193442+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:36:46.193442+0000","last_undegraded":"2024-09-17T05:36:46.193442+0000","last_fullsized":"2024-09-17T05:36:46.193442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-09-17T05:36:46.193342+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-09-17T05:36:46.193342+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.193342+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:55.328039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.662759+0000","last_change":"2024-09-17T05:36:45.662759+0000","last_active":"2024-09-17T05:36:45.662759+0000","last_peered":"2024-09-17T05:36:45.662759+0000","last_clean":"2024-09-17T05:36:45.662759+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:36:45.662759+0000","last_undegraded":"2024-09-17T05:36:45.662759+0000","last_fullsized":"2024-09-17T05:36:45.662759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-09-17T05:36:45.662651+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-09-17T05:36:45.662651+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.662651+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:13.241217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"31'5405","reported_seq":5457,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.081958+0000","last_change":"2024-09-17T05:36:46.081958+0000","last_active":"2024-09-17T05:36:46.081958+0000","last_peered":"2024-09-17T05:36:46.081958+0000","last_clean":"2024-09-17T05:36:46.081958+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:36:46.081958+0000","last_undegraded":"2024-09-17T05:36:46.081958+0000","last_fullsized":"2024-09-17T05:36:46.081958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5405","last_scrub_stamp":"2024-09-17T05:36:46.081821+0000","last_deep_scrub":"31'5405","last_deep_scrub_stamp":"2024-09-17T05:36:46.081821+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.081821+0000","objects_scrubbed":0,"log_size":5405,"log_dups_size":0,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:48.866580+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":0,"num_write":79725,"num_write_kb":1002963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.356509+0000","last_change":"2024-09-17T05:36:39.356509+0000","last_active":"2024-09-17T05:36:39.356509+0000","last_peered":"2024-09-17T05:36:39.356509+0000","last_clean":"2024-09-17T05:36:39.356509+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:36:39.356509+0000","last_undegraded":"2024-09-17T05:36:39.356509+0000","last_fullsized":"2024-09-17T05:36:39.356509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:39.356424+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:39.356424+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.356424+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:44.278824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"31'14","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.739166+0000","last_change":"2024-09-17T05:36:41.739166+0000","last_active":"2024-09-17T05:36:41.739166+0000","last_peered":"2024-09-17T05:36:41.739166+0000","last_clean":"2024-09-17T05:36:41.739166+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:36:41.739166+0000","last_undegraded":"2024-09-17T05:36:41.739166+0000","last_fullsized":"2024-09-17T05:36:41.739166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:41.739054+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:41.739054+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.739054+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:40.946191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'5609","reported_seq":5665,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:54.671761+0000","last_change":"2024-09-17T05:36:54.671761+0000","last_active":"2024-09-17T05:36:54.671761+0000","last_peered":"2024-09-17T05:36:54.671761+0000","last_clean":"2024-09-17T05:36:54.671761+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:36:54.671761+0000","last_undegraded":"2024-09-17T05:36:54.671761+0000","last_fullsized":"2024-09-17T05:36:54.671761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5609","last_scrub_stamp":"2024-09-17T05:36:54.671595+0000","last_deep_scrub":"31'5609","last_deep_scrub_stamp":"2024-09-17T05:36:54.671595+0000","last_clean_scrub_stamp":"2024-09-17T05:36:54.671595+0000","objects_scrubbed":0,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:15.270945+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82713,"num_write_kb":1020969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6490","reported_seq":6546,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.670705+0000","last_change":"2024-09-17T05:36:43.670705+0000","last_active":"2024-09-17T05:36:43.670705+0000","last_peered":"2024-09-17T05:36:43.670705+0000","last_clean":"2024-09-17T05:36:43.670705+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:36:43.670705+0000","last_undegraded":"2024-09-17T05:36:43.670705+0000","last_fullsized":"2024-09-17T05:36:43.670705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6490","last_scrub_stamp":"2024-09-17T05:36:43.670588+0000","last_deep_scrub":"31'6490","last_deep_scrub_stamp":"2024-09-17T05:36:43.670588+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.670588+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:52.614622+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":95953,"num_write_kb":1202164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.618805+0000","last_change":"2024-09-17T05:36:53.618805+0000","last_active":"2024-09-17T05:36:53.618805+0000","last_peered":"2024-09-17T05:36:53.618805+0000","last_clean":"2024-09-17T05:36:53.618805+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:36:53.618805+0000","last_undegraded":"2024-09-17T05:36:53.618805+0000","last_fullsized":"2024-09-17T05:36:53.618805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:53.618649+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:53.618649+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.618649+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:18.914369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"31'31","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.521707+0000","last_change":"2024-09-17T05:36:47.521707+0000","last_active":"2024-09-17T05:36:47.521707+0000","last_peered":"2024-09-17T05:36:47.521707+0000","last_clean":"2024-09-17T05:36:47.521707+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:36:47.521707+0000","last_undegraded":"2024-09-17T05:36:47.521707+0000","last_fullsized":"2024-09-17T05:36:47.521707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'31","last_scrub_stamp":"2024-09-17T05:36:47.521587+0000","last_deep_scrub":"31'31","last_deep_scrub_stamp":"2024-09-17T05:36:47.521587+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.521587+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:56.095889+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":25,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"31'5638","reported_seq":5698,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.463714+0000","last_change":"2024-09-17T05:36:39.463714+0000","last_active":"2024-09-17T05:36:39.463714+0000","last_peered":"2024-09-17T05:36:39.463714+0000","last_clean":"2024-09-17T05:36:39.463714+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:36:39.463714+0000","last_undegraded":"2024-09-17T05:36:39.463714+0000","last_fullsized":"2024-09-17T05:36:39.463714+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5638","last_scrub_stamp":"2024-09-17T05:36:39.463599+0000","last_deep_scrub":"31'5638","last_deep_scrub_stamp":"2024-09-17T05:36:39.463599+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.463599+0000","objects_scrubbed":0,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:17.221508+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":83062,"num_write_kb":1044536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"31'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.224277+0000","last_change":"2024-09-17T05:36:45.224277+0000","last_active":"2024-09-17T05:36:45.224277+0000","last_peered":"2024-09-17T05:36:45.224277+0000","last_clean":"2024-09-17T05:36:45.224277+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:36:45.224277+0000","last_undegraded":"2024-09-17T05:36:45.224277+0000","last_fullsized":"2024-09-17T05:36:45.224277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:45.224183+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:45.224183+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.224183+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:59.844340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"31'7","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.039099+0000","last_change":"2024-09-17T05:36:39.039099+0000","last_active":"2024-09-17T05:36:39.039099+0000","last_peered":"2024-09-17T05:36:39.039099+0000","last_clean":"2024-09-17T05:36:39.039099+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:36:39.039099+0000","last_undegraded":"2024-09-17T05:36:39.039099+0000","last_fullsized":"2024-09-17T05:36:39.039099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:39.039005+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:39.039005+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.039005+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:12:41.557964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"31'23","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.040169+0000","last_change":"2024-09-17T05:27:25.654938+0000","last_active":"2024-09-17T05:36:38.040169+0000","last_peered":"2024-09-17T05:36:38.040169+0000","last_clean":"2024-09-17T05:36:38.040169+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:36:38.040169+0000","last_undegraded":"2024-09-17T05:36:38.040169+0000","last_fullsized":"2024-09-17T05:36:38.040169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:24.623530+0000","last_clean_scrub_stamp":"2024-09-17T05:27:24.623530+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":4108,"num_write":18,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.643978+0000","last_change":"2024-09-17T05:36:47.643978+0000","last_active":"2024-09-17T05:36:47.643978+0000","last_peered":"2024-09-17T05:36:47.643978+0000","last_clean":"2024-09-17T05:36:47.643978+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:36:47.643978+0000","last_undegraded":"2024-09-17T05:36:47.643978+0000","last_fullsized":"2024-09-17T05:36:47.643978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:47.643882+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:47.643882+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.643882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:33.604932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4609","reported_seq":4655,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.622729+0000","last_change":"2024-09-17T05:36:39.622729+0000","last_active":"2024-09-17T05:36:39.622729+0000","last_peered":"2024-09-17T05:36:39.622729+0000","last_clean":"2024-09-17T05:36:39.622729+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:36:39.622729+0000","last_undegraded":"2024-09-17T05:36:39.622729+0000","last_fullsized":"2024-09-17T05:36:39.622729+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4609","last_scrub_stamp":"2024-09-17T05:36:39.622630+0000","last_deep_scrub":"31'4609","last_deep_scrub_stamp":"2024-09-17T05:36:39.622630+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.622630+0000","objects_scrubbed":0,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:54.799297+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":68113,"num_write_kb":833128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"31'1238","reported_seq":1287,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.212158+0000","last_change":"2024-09-17T05:36:39.212158+0000","last_active":"2024-09-17T05:36:39.212158+0000","last_peered":"2024-09-17T05:36:39.212158+0000","last_clean":"2024-09-17T05:36:39.212158+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:36:39.212158+0000","last_undegraded":"2024-09-17T05:36:39.212158+0000","last_fullsized":"2024-09-17T05:36:39.212158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'1238","last_scrub_stamp":"2024-09-17T05:36:39.212032+0000","last_deep_scrub":"31'1238","last_deep_scrub_stamp":"2024-09-17T05:36:39.212032+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.212032+0000","objects_scrubbed":3,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:13.037156+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4196027,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1229,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"31'13","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.706309+0000","last_change":"2024-09-17T05:36:38.706309+0000","last_active":"2024-09-17T05:36:38.706309+0000","last_peered":"2024-09-17T05:36:38.706309+0000","last_clean":"2024-09-17T05:36:38.706309+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:36:38.706309+0000","last_undegraded":"2024-09-17T05:36:38.706309+0000","last_fullsized":"2024-09-17T05:36:38.706309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:38.706208+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:38.706208+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.706208+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:59.688655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'6242","reported_seq":6299,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.027356+0000","last_change":"2024-09-17T05:36:49.027356+0000","last_active":"2024-09-17T05:36:49.027356+0000","last_peered":"2024-09-17T05:36:49.027356+0000","last_clean":"2024-09-17T05:36:49.027356+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:36:49.027356+0000","last_undegraded":"2024-09-17T05:36:49.027356+0000","last_fullsized":"2024-09-17T05:36:49.027356+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6242","last_scrub_stamp":"2024-09-17T05:36:49.027241+0000","last_deep_scrub":"31'6242","last_deep_scrub_stamp":"2024-09-17T05:36:49.027241+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.027241+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:59:05.514832+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":92301,"num_write_kb":1140422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.338972+0000","last_change":"2024-09-17T05:36:47.338972+0000","last_active":"2024-09-17T05:36:47.338972+0000","last_peered":"2024-09-17T05:36:47.338972+0000","last_clean":"2024-09-17T05:36:47.338972+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:36:47.338972+0000","last_undegraded":"2024-09-17T05:36:47.338972+0000","last_fullsized":"2024-09-17T05:36:47.338972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:47.338887+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:47.338887+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.338887+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:40.881120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"31'13","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.015447+0000","last_change":"2024-09-17T05:36:42.015447+0000","last_active":"2024-09-17T05:36:42.015447+0000","last_peered":"2024-09-17T05:36:42.015447+0000","last_clean":"2024-09-17T05:36:42.015447+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:36:42.015447+0000","last_undegraded":"2024-09-17T05:36:42.015447+0000","last_fullsized":"2024-09-17T05:36:42.015447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:42.015349+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:42.015349+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.015349+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:30.039576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"31'15","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.931939+0000","last_change":"2024-09-17T05:27:26.664524+0000","last_active":"2024-09-17T05:36:37.931939+0000","last_peered":"2024-09-17T05:36:37.931939+0000","last_clean":"2024-09-17T05:36:37.931939+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:36:37.931939+0000","last_undegraded":"2024-09-17T05:36:37.931939+0000","last_fullsized":"2024-09-17T05:36:37.931939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T05:27:25.629709+0000","last_clean_scrub_stamp":"2024-09-17T05:27:25.629709+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.177392+0000","last_change":"2024-09-17T05:36:48.177392+0000","last_active":"2024-09-17T05:36:48.177392+0000","last_peered":"2024-09-17T05:36:48.177392+0000","last_clean":"2024-09-17T05:36:48.177392+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:36:48.177392+0000","last_undegraded":"2024-09-17T05:36:48.177392+0000","last_fullsized":"2024-09-17T05:36:48.177392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:48.177304+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:48.177304+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.177304+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:34:49.273171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.174036+0000","last_change":"2024-09-17T05:36:48.174036+0000","last_active":"2024-09-17T05:36:48.174036+0000","last_peered":"2024-09-17T05:36:48.174036+0000","last_clean":"2024-09-17T05:36:48.174036+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:36:48.174036+0000","last_undegraded":"2024-09-17T05:36:48.174036+0000","last_fullsized":"2024-09-17T05:36:48.174036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:48.173944+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:48.173944+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.173944+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:57.117978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.114788+0000","last_change":"2024-09-17T05:36:43.114788+0000","last_active":"2024-09-17T05:36:43.114788+0000","last_peered":"2024-09-17T05:36:43.114788+0000","last_clean":"2024-09-17T05:36:43.114788+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:36:43.114788+0000","last_undegraded":"2024-09-17T05:36:43.114788+0000","last_fullsized":"2024-09-17T05:36:43.114788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:43.114691+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:43.114691+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.114691+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:50.140756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.392372+0000","last_change":"2024-09-17T05:36:40.392372+0000","last_active":"2024-09-17T05:36:40.392372+0000","last_peered":"2024-09-17T05:36:40.392372+0000","last_clean":"2024-09-17T05:36:40.392372+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:36:40.392372+0000","last_undegraded":"2024-09-17T05:36:40.392372+0000","last_fullsized":"2024-09-17T05:36:40.392372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:40.392280+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:40.392280+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.392280+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:53:44.758222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.252032+0000","last_change":"2024-09-17T05:36:50.252032+0000","last_active":"2024-09-17T05:36:50.252032+0000","last_peered":"2024-09-17T05:36:50.252032+0000","last_clean":"2024-09-17T05:36:50.252032+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:36:50.252032+0000","last_undegraded":"2024-09-17T05:36:50.252032+0000","last_fullsized":"2024-09-17T05:36:50.252032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:50.251929+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:50.251929+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.251929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:34.342583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"31'12","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.462043+0000","last_change":"2024-09-17T05:36:48.462043+0000","last_active":"2024-09-17T05:36:48.462043+0000","last_peered":"2024-09-17T05:36:48.462043+0000","last_clean":"2024-09-17T05:36:48.462043+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:36:48.462043+0000","last_undegraded":"2024-09-17T05:36:48.462043+0000","last_fullsized":"2024-09-17T05:36:48.462043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:48.461953+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:48.461953+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.461953+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:12.821365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.204218+0000","last_change":"2024-09-17T05:36:49.204218+0000","last_active":"2024-09-17T05:36:49.204218+0000","last_peered":"2024-09-17T05:36:49.204218+0000","last_clean":"2024-09-17T05:36:49.204218+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:36:49.204218+0000","last_undegraded":"2024-09-17T05:36:49.204218+0000","last_fullsized":"2024-09-17T05:36:49.204218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:49.204112+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:49.204112+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.204112+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:08.061956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"31'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.730554+0000","last_change":"2024-09-17T05:36:39.730554+0000","last_active":"2024-09-17T05:36:39.730554+0000","last_peered":"2024-09-17T05:36:39.730554+0000","last_clean":"2024-09-17T05:36:39.730554+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:36:39.730554+0000","last_undegraded":"2024-09-17T05:36:39.730554+0000","last_fullsized":"2024-09-17T05:36:39.730554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:39.730431+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:39.730431+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.730431+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:40.179412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"31'17","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.684772+0000","last_change":"2024-09-17T05:36:48.684772+0000","last_active":"2024-09-17T05:36:48.684772+0000","last_peered":"2024-09-17T05:36:48.684772+0000","last_clean":"2024-09-17T05:36:48.684772+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:36:48.684772+0000","last_undegraded":"2024-09-17T05:36:48.684772+0000","last_fullsized":"2024-09-17T05:36:48.684772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'17","last_scrub_stamp":"2024-09-17T05:36:48.684691+0000","last_deep_scrub":"31'17","last_deep_scrub_stamp":"2024-09-17T05:36:48.684691+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.684691+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:15.592178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.140037+0000","last_change":"2024-09-17T05:36:37.610072+0000","last_active":"2024-09-17T05:36:38.140037+0000","last_peered":"2024-09-17T05:36:38.140037+0000","last_clean":"2024-09-17T05:36:38.140037+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:36:38.140037+0000","last_undegraded":"2024-09-17T05:36:38.140037+0000","last_fullsized":"2024-09-17T05:36:38.140037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:37.609959+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:37.609959+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.609959+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:03.953462+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:54.412079+0000","last_change":"2024-09-17T05:36:54.412079+0000","last_active":"2024-09-17T05:36:54.412079+0000","last_peered":"2024-09-17T05:36:54.412079+0000","last_clean":"2024-09-17T05:36:54.412079+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:36:54.412079+0000","last_undegraded":"2024-09-17T05:36:54.412079+0000","last_fullsized":"2024-09-17T05:36:54.412079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:54.411925+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:54.411925+0000","last_clean_scrub_stamp":"2024-09-17T05:36:54.411925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:44.212757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"31'19","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.004268+0000","last_change":"2024-09-17T05:36:45.004268+0000","last_active":"2024-09-17T05:36:45.004268+0000","last_peered":"2024-09-17T05:36:45.004268+0000","last_clean":"2024-09-17T05:36:45.004268+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:36:45.004268+0000","last_undegraded":"2024-09-17T05:36:45.004268+0000","last_fullsized":"2024-09-17T05:36:45.004268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:45.004163+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:45.004163+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.004163+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:00:37.747058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"31'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.390514+0000","last_change":"2024-09-17T05:36:38.390411+0000","last_active":"2024-09-17T05:36:38.390514+0000","last_peered":"2024-09-17T05:36:38.390514+0000","last_clean":"2024-09-17T05:36:38.390514+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:36:38.390514+0000","last_undegraded":"2024-09-17T05:36:38.390514+0000","last_fullsized":"2024-09-17T05:36:38.390514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:38.390300+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:38.390300+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.390300+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:21.127766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"31'12","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.462921+0000","last_change":"2024-09-17T05:36:49.462851+0000","last_active":"2024-09-17T05:36:49.462921+0000","last_peered":"2024-09-17T05:36:49.462921+0000","last_clean":"2024-09-17T05:36:49.462921+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:36:49.462921+0000","last_undegraded":"2024-09-17T05:36:49.462921+0000","last_fullsized":"2024-09-17T05:36:49.462921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:49.462764+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:49.462764+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.462764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:53.324712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"31'14","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.290552+0000","last_change":"2024-09-17T05:36:51.290486+0000","last_active":"2024-09-17T05:36:51.290552+0000","last_peered":"2024-09-17T05:36:51.290552+0000","last_clean":"2024-09-17T05:36:51.290552+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:36:51.290552+0000","last_undegraded":"2024-09-17T05:36:51.290552+0000","last_fullsized":"2024-09-17T05:36:51.290552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:51.290399+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:51.290399+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.290399+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:44:51.991292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"31'17","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.119537+0000","last_change":"2024-09-17T05:36:46.119431+0000","last_active":"2024-09-17T05:36:46.119537+0000","last_peered":"2024-09-17T05:36:46.119537+0000","last_clean":"2024-09-17T05:36:46.119537+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:36:46.119537+0000","last_undegraded":"2024-09-17T05:36:46.119537+0000","last_fullsized":"2024-09-17T05:36:46.119537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'17","last_scrub_stamp":"2024-09-17T05:36:46.119291+0000","last_deep_scrub":"31'17","last_deep_scrub_stamp":"2024-09-17T05:36:46.119291+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.119291+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:15.011798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"31'15","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.147113+0000","last_change":"2024-09-17T05:36:47.147113+0000","last_active":"2024-09-17T05:36:47.147113+0000","last_peered":"2024-09-17T05:36:47.147113+0000","last_clean":"2024-09-17T05:36:47.147113+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:36:47.147113+0000","last_undegraded":"2024-09-17T05:36:47.147113+0000","last_fullsized":"2024-09-17T05:36:47.147113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:47.147028+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:47.147028+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.147028+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:19.880014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"31'10","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.610809+0000","last_change":"2024-09-17T05:36:38.610809+0000","last_active":"2024-09-17T05:36:38.610809+0000","last_peered":"2024-09-17T05:36:38.610809+0000","last_clean":"2024-09-17T05:36:38.610809+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:36:38.610809+0000","last_undegraded":"2024-09-17T05:36:38.610809+0000","last_fullsized":"2024-09-17T05:36:38.610809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:38.610732+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:38.610732+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.610732+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:32.152965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"31'12","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.675441+0000","last_change":"2024-09-17T05:36:42.675441+0000","last_active":"2024-09-17T05:36:42.675441+0000","last_peered":"2024-09-17T05:36:42.675441+0000","last_clean":"2024-09-17T05:36:42.675441+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:36:42.675441+0000","last_undegraded":"2024-09-17T05:36:42.675441+0000","last_fullsized":"2024-09-17T05:36:42.675441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:42.675339+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:42.675339+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.675339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:09.582826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"31'13","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.696505+0000","last_change":"2024-09-17T05:36:46.696505+0000","last_active":"2024-09-17T05:36:46.696505+0000","last_peered":"2024-09-17T05:36:46.696505+0000","last_clean":"2024-09-17T05:36:46.696505+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:36:46.696505+0000","last_undegraded":"2024-09-17T05:36:46.696505+0000","last_fullsized":"2024-09-17T05:36:46.696505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:46.696388+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:46.696388+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.696388+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:09.445934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"31'16","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.760807+0000","last_change":"2024-09-17T05:36:42.760807+0000","last_active":"2024-09-17T05:36:42.760807+0000","last_peered":"2024-09-17T05:36:42.760807+0000","last_clean":"2024-09-17T05:36:42.760807+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:36:42.760807+0000","last_undegraded":"2024-09-17T05:36:42.760807+0000","last_fullsized":"2024-09-17T05:36:42.760807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-09-17T05:36:42.760720+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-09-17T05:36:42.760720+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.760720+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:30.784948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"31'5","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.310258+0000","last_change":"2024-09-17T05:36:52.310258+0000","last_active":"2024-09-17T05:36:52.310258+0000","last_peered":"2024-09-17T05:36:52.310258+0000","last_clean":"2024-09-17T05:36:52.310258+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:36:52.310258+0000","last_undegraded":"2024-09-17T05:36:52.310258+0000","last_fullsized":"2024-09-17T05:36:52.310258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-09-17T05:36:52.310114+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-09-17T05:36:52.310114+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.310114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:39.798350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"31'9","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.661839+0000","last_change":"2024-09-17T05:36:44.661839+0000","last_active":"2024-09-17T05:36:44.661839+0000","last_peered":"2024-09-17T05:36:44.661839+0000","last_clean":"2024-09-17T05:36:44.661839+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:36:44.661839+0000","last_undegraded":"2024-09-17T05:36:44.661839+0000","last_fullsized":"2024-09-17T05:36:44.661839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:44.661741+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:44.661741+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.661741+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:05.214263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.384149+0000","last_change":"2024-09-17T05:36:42.384149+0000","last_active":"2024-09-17T05:36:42.384149+0000","last_peered":"2024-09-17T05:36:42.384149+0000","last_clean":"2024-09-17T05:36:42.384149+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:36:42.384149+0000","last_undegraded":"2024-09-17T05:36:42.384149+0000","last_fullsized":"2024-09-17T05:36:42.384149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:42.384012+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:42.384012+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.384012+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:57.560680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"31'15","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.645068+0000","last_change":"2024-09-17T05:36:48.645068+0000","last_active":"2024-09-17T05:36:48.645068+0000","last_peered":"2024-09-17T05:36:48.645068+0000","last_clean":"2024-09-17T05:36:48.645068+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:36:48.645068+0000","last_undegraded":"2024-09-17T05:36:48.645068+0000","last_fullsized":"2024-09-17T05:36:48.645068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:48.644964+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:48.644964+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.644964+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:09.972427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"31'7","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.348905+0000","last_change":"2024-09-17T05:36:41.348905+0000","last_active":"2024-09-17T05:36:41.348905+0000","last_peered":"2024-09-17T05:36:41.348905+0000","last_clean":"2024-09-17T05:36:41.348905+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:36:41.348905+0000","last_undegraded":"2024-09-17T05:36:41.348905+0000","last_fullsized":"2024-09-17T05:36:41.348905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:41.348751+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:41.348751+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.348751+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:08.318810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.316305+0000","last_change":"2024-09-17T05:36:53.316305+0000","last_active":"2024-09-17T05:36:53.316305+0000","last_peered":"2024-09-17T05:36:53.316305+0000","last_clean":"2024-09-17T05:36:53.316305+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:36:53.316305+0000","last_undegraded":"2024-09-17T05:36:53.316305+0000","last_fullsized":"2024-09-17T05:36:53.316305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:53.316209+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:53.316209+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.316209+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:42:23.795879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"31'6","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:54.353964+0000","last_change":"2024-09-17T05:36:54.353964+0000","last_active":"2024-09-17T05:36:54.353964+0000","last_peered":"2024-09-17T05:36:54.353964+0000","last_clean":"2024-09-17T05:36:54.353964+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:36:54.353964+0000","last_undegraded":"2024-09-17T05:36:54.353964+0000","last_fullsized":"2024-09-17T05:36:54.353964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-09-17T05:36:54.353868+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-09-17T05:36:54.353868+0000","last_clean_scrub_stamp":"2024-09-17T05:36:54.353868+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:03:03.509468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"31'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.004419+0000","last_change":"2024-09-17T05:36:44.004419+0000","last_active":"2024-09-17T05:36:44.004419+0000","last_peered":"2024-09-17T05:36:44.004419+0000","last_clean":"2024-09-17T05:36:44.004419+0000","last_became_active":"2024-09-17T05:28:07.483003+0000","last_became_peered":"2024-09-17T05:28:07.483003+0000","last_unstale":"2024-09-17T05:36:44.004419+0000","last_undegraded":"2024-09-17T05:36:44.004419+0000","last_fullsized":"2024-09-17T05:36:44.004419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:44.004318+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:44.004318+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.004318+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:02.130745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":0,"num_write":2621864,"num_write_kb":32721427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177787,"ondisk_log_size":177787,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":82273271,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":349,"num_read_kb":78313,"num_write":2736,"num_write_kb":82427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15000,"num_omap_keys":30,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34103296,"data_stored":164546542,"data_compressed":18858484,"data_compressed_allocated":21905408,"data_compressed_original":152502272,"omap_allocated":0,"internal_metadata":0},"log_size":2703,"ondisk_log_size":2703,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509556,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215208,"kb_used_data":5976,"kb_used_omap":1,"kb_used_meta":1209214,"kb_avail":93156632,"statfs":{"total":96636764160,"available":95392391168,"internally_reserved":0,"allocated":6119424,"data_stored":37895144,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":1593,"internal_metadata":1238235591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509558,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":844032,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":840830,"kb_avail":93527808,"statfs":{"total":96636764160,"available":95772475392,"internally_reserved":0,"allocated":3260416,"data_stored":12727699,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":1595,"internal_metadata":861010373},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509558,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1132012,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":1127486,"kb_avail":93239828,"statfs":{"total":96636764160,"available":95477583872,"internally_reserved":0,"allocated":4616192,"data_stored":24186871,"data_compressed":2785195,"data_compressed_allocated":3256320,"data_compressed_original":23027712,"omap_allocated":1595,"internal_metadata":1154546117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509559,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736580,"kb_used_data":5236,"kb_used_omap":1,"kb_used_meta":731326,"kb_avail":93635260,"statfs":{"total":96636764160,"available":95882506240,"internally_reserved":0,"allocated":5361664,"data_stored":27793064,"data_compressed":3270621,"data_compressed_allocated":3801088,"data_compressed_original":26439680,"omap_allocated":1595,"internal_metadata":748878277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542260,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129504,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":1127038,"kb_avail":93242336,"statfs":{"total":96636764160,"available":95480152064,"internally_reserved":0,"allocated":2506752,"data_stored":12822170,"data_compressed":1499845,"data_compressed_allocated":1761280,"data_compressed_original":12292096,"omap_allocated":1590,"internal_metadata":1154087370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542261,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1086856,"kb_used_data":4664,"kb_used_omap":1,"kb_used_meta":1082174,"kb_avail":93284984,"statfs":{"total":96636764160,"available":95523823616,"internally_reserved":0,"allocated":4775936,"data_stored":12729298,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":1594,"internal_metadata":1108146630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542260,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":916820,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":913726,"kb_avail":93455020,"statfs":{"total":96636764160,"available":95697940480,"internally_reserved":0,"allocated":3149824,"data_stored":12729233,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":1594,"internal_metadata":935655878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574966,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":880292,"kb_used_data":6932,"kb_used_omap":1,"kb_used_meta":873342,"kb_avail":93491548,"statfs":{"total":96636764160,"available":95735345152,"internally_reserved":0,"allocated":7098368,"data_stored":25996447,"data_compressed":2718385,"data_compressed_allocated":3203072,"data_compressed_original":22315008,"omap_allocated":1595,"internal_metadata":894302661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5029888,"data_stored":27648983,"data_compressed":3270621,"data_compressed_allocated":3801088,"data_compressed_original":26439680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":23452422,"data_compressed":2752708,"data_compressed_allocated":3211264,"data_compressed_original":22454272,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":12583618,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":37751063,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":12585152,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6701056,"data_stored":25261998,"data_compressed":2685898,"data_compressed_allocated":3158016,"data_compressed_original":21741568,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":12585217,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":12678089,"data_compressed":1499845,"data_compressed_allocated":1761280,"data_compressed_original":12292096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:37:00.125 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.125 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.125 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.126 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.126 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.126 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.126 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.127 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-17T05:27:29.653148+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.127 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-17T05:27:24.623530+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.127 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-17T05:27:25.629709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=36, tm_sec=31, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:37:00.128 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T05:37:20.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 -- 172.21.15.117:0/3989068037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7340cad30 msgr2=0x7fd7340c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 --2- 172.21.15.117:0/3989068037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7340cad30 0x7fd7340c37c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009830 tx=0x7fd72c02a510 comp rx=0 tx=0).stop 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 -- 172.21.15.117:0/3989068037 shutdown_connections 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 --2- 172.21.15.117:0/3989068037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7340cad30 0x7fd7340c37c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 -- 172.21.15.117:0/3989068037 >> 172.21.15.117:0/3989068037 conn(0x7fd734076430 msgr2=0x7fd734072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 -- 172.21.15.117:0/3989068037 shutdown_connections 2024-09-17T05:37:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.245+0000 7fd7389c3640 1 -- 172.21.15.117:0/3989068037 wait complete. 2024-09-17T05:37:20.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.247+0000 7fd7389c3640 1 Processor -- start 2024-09-17T05:37:20.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.247+0000 7fd7389c3640 1 -- start start 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd7389c3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd7340cad30 0x7fd73415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd7389c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 0x7fd734147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd7389c3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd73415d220 0x7fd734156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd7389c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd7340cbc30 con 0x7fd734153260 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd7389c3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd73415cd00 con 0x7fd73415d220 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd7389c3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd7340c9e40 con 0x7fd7340cad30 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 0x7fd734147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 0x7fd734147480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51772/0 (socket says 172.21.15.117:51772) 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 -- 172.21.15.117:0/45556496 learned_addr learned my addr 172.21.15.117:0/45556496 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd732d9d640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd73415d220 0x7fd734156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd73259c640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd7340cad30 0x7fd73415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 -- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd7340cad30 msgr2=0x7fd73415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd7340cad30 0x7fd73415bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 -- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd73415d220 msgr2=0x7fd734156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd73415d220 0x7fd734156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.248+0000 7fd731d9b640 1 -- 172.21.15.117:0/45556496 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73415c4a0 con 0x7fd734153260 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.249+0000 7fd731d9b640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 0x7fd734147480 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c7d0 tx=0x7fd72800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.249+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728017070 con 0x7fd734153260 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.249+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd728005070 con 0x7fd734153260 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.249+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7340ce550 con 0x7fd734153260 2024-09-17T05:37:20.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.249+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280054a0 con 0x7fd734153260 2024-09-17T05:37:20.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.249+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7340cca00 con 0x7fd734153260 2024-09-17T05:37:20.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.250+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7340c8a90 con 0x7fd734153260 2024-09-17T05:37:20.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.250+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd72801a070 con 0x7fd734153260 2024-09-17T05:37:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.251+0000 7fd722ffd640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd714078b90 0x7fd71407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.251+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fd72809a7e0 con 0x7fd734153260 2024-09-17T05:37:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.251+0000 7fd73259c640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd714078b90 0x7fd71407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.252+0000 7fd73259c640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd714078b90 0x7fd71407b0a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd72c034000 tx=0x7fd72c004730 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:20.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.256+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd728062190 con 0x7fd734153260 2024-09-17T05:37:20.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.431+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 --> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7340ca300 con 0x7fd714078b90 2024-09-17T05:37:20.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.437+0000 7fd722ffd640 1 -- 172.21.15.117:0/45556496 <== mgr.4114 v2:172.21.15.119:6800/61506279 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+335090 (secure 0 0 0) 0x7fd7340ca300 con 0x7fd714078b90 2024-09-17T05:37:20.442 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:37:20.444 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-17T05:37:20.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd714078b90 msgr2=0x7fd71407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd714078b90 0x7fd71407b0a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd72c034000 tx=0x7fd72c004730 comp rx=0 tx=0).stop 2024-09-17T05:37:20.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 msgr2=0x7fd734147480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 0x7fd734147480 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c7d0 tx=0x7fd72800cc90 comp rx=0 tx=0).stop 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 shutdown_connections 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd714078b90 0x7fd71407b0a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd73415d220 0x7fd734156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd734153260 0x7fd734147480 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 --2- 172.21.15.117:0/45556496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd7340cad30 0x7fd73415bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 >> 172.21.15.117:0/45556496 conn(0x7fd734076430 msgr2=0x7fd734149430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 shutdown_connections 2024-09-17T05:37:20.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.441+0000 7fd7389c3640 1 -- 172.21.15.117:0/45556496 wait complete. 2024-09-17T05:37:20.453 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-17T05:37:19.544429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":82863639,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":940,"num_read_kb":78368,"num_write":2624701,"num_write_kb":32805482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15000,"num_omap_keys":30,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181343,"ondisk_log_size":181343,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7941304,"kb_used_data":36024,"kb_used_omap":12,"kb_used_meta":7905139,"kb_avail":747033416,"statfs":{"total":773094113280,"available":764962217984,"internally_reserved":0,"allocated":36888576,"data_stored":166879926,"data_compressed":18923458,"data_compressed_allocated":21995520,"data_compressed_original":153649152,"omap_allocated":12751,"internal_metadata":8094862897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003705"},"pg_stats":[{"pgid":"2.1f","version":"31'23","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.667369+0000","last_change":"2024-09-17T05:36:44.667369+0000","last_active":"2024-09-17T05:36:44.667369+0000","last_peered":"2024-09-17T05:36:44.667369+0000","last_clean":"2024-09-17T05:36:44.667369+0000","last_became_active":"2024-09-17T05:27:25.653046+0000","last_became_peered":"2024-09-17T05:27:25.653046+0000","last_unstale":"2024-09-17T05:36:44.667369+0000","last_undegraded":"2024-09-17T05:36:44.667369+0000","last_fullsized":"2024-09-17T05:36:44.667369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-09-17T05:36:44.667275+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-09-17T05:36:44.667275+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.667275+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:19:02.458753+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":4124,"num_write":15,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.211543+0000","last_change":"2024-09-17T05:36:43.211543+0000","last_active":"2024-09-17T05:36:43.211543+0000","last_peered":"2024-09-17T05:36:43.211543+0000","last_clean":"2024-09-17T05:36:43.211543+0000","last_became_active":"2024-09-17T05:27:26.656330+0000","last_became_peered":"2024-09-17T05:27:26.656330+0000","last_unstale":"2024-09-17T05:36:43.211543+0000","last_undegraded":"2024-09-17T05:36:43.211543+0000","last_fullsized":"2024-09-17T05:36:43.211543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-09-17T05:36:43.211450+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-09-17T05:36:43.211450+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.211450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:39:27.204598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"31'5864","reported_seq":5919,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.052321+0000","last_change":"2024-09-17T05:36:48.052321+0000","last_active":"2024-09-17T05:36:48.052321+0000","last_peered":"2024-09-17T05:36:48.052321+0000","last_clean":"2024-09-17T05:36:48.052321+0000","last_became_active":"2024-09-17T05:27:30.686343+0000","last_became_peered":"2024-09-17T05:27:30.686343+0000","last_unstale":"2024-09-17T05:36:48.052321+0000","last_undegraded":"2024-09-17T05:36:48.052321+0000","last_fullsized":"2024-09-17T05:36:48.052321+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5864","last_scrub_stamp":"2024-09-17T05:36:48.052212+0000","last_deep_scrub":"31'5864","last_deep_scrub_stamp":"2024-09-17T05:36:48.052212+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.052212+0000","objects_scrubbed":0,"log_size":5864,"log_dups_size":0,"ondisk_log_size":5864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:56.791747+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":86661,"num_write_kb":1098753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4,1],"acting":[7,6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"31'28","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.255358+0000","last_change":"2024-09-17T05:36:42.255358+0000","last_active":"2024-09-17T05:36:42.255358+0000","last_peered":"2024-09-17T05:36:42.255358+0000","last_clean":"2024-09-17T05:36:42.255358+0000","last_became_active":"2024-09-17T05:27:25.653281+0000","last_became_peered":"2024-09-17T05:27:25.653281+0000","last_unstale":"2024-09-17T05:36:42.255358+0000","last_undegraded":"2024-09-17T05:36:42.255358+0000","last_fullsized":"2024-09-17T05:36:42.255358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'28","last_scrub_stamp":"2024-09-17T05:36:42.255268+0000","last_deep_scrub":"31'28","last_deep_scrub_stamp":"2024-09-17T05:36:42.255268+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.255268+0000","objects_scrubbed":1,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:58.452012+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":19,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.181380+0000","last_change":"2024-09-17T05:36:47.181380+0000","last_active":"2024-09-17T05:36:47.181380+0000","last_peered":"2024-09-17T05:36:47.181380+0000","last_clean":"2024-09-17T05:36:47.181380+0000","last_became_active":"2024-09-17T05:27:26.658388+0000","last_became_peered":"2024-09-17T05:27:26.658388+0000","last_unstale":"2024-09-17T05:36:47.181380+0000","last_undegraded":"2024-09-17T05:36:47.181380+0000","last_fullsized":"2024-09-17T05:36:47.181380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:47.181292+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:47.181292+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.181292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:10.154989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"31'5772","reported_seq":5829,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.347283+0000","last_change":"2024-09-17T05:36:48.347203+0000","last_active":"2024-09-17T05:36:48.347283+0000","last_peered":"2024-09-17T05:36:48.347283+0000","last_clean":"2024-09-17T05:36:48.347283+0000","last_became_active":"2024-09-17T05:27:30.677132+0000","last_became_peered":"2024-09-17T05:27:30.677132+0000","last_unstale":"2024-09-17T05:36:48.347283+0000","last_undegraded":"2024-09-17T05:36:48.347283+0000","last_fullsized":"2024-09-17T05:36:48.347283+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5772","last_scrub_stamp":"2024-09-17T05:36:48.347064+0000","last_deep_scrub":"31'5772","last_deep_scrub_stamp":"2024-09-17T05:36:48.347064+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.347064+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:57.310584+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":85101,"num_write_kb":1045061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4,6],"acting":[2,7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"31'57","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.645973+0000","last_change":"2024-09-17T05:36:40.645973+0000","last_active":"2024-09-17T05:36:40.645973+0000","last_peered":"2024-09-17T05:36:40.645973+0000","last_clean":"2024-09-17T05:36:40.645973+0000","last_became_active":"2024-09-17T05:27:25.659634+0000","last_became_peered":"2024-09-17T05:27:25.659634+0000","last_unstale":"2024-09-17T05:36:40.645973+0000","last_undegraded":"2024-09-17T05:36:40.645973+0000","last_fullsized":"2024-09-17T05:36:40.645973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'57","last_scrub_stamp":"2024-09-17T05:36:40.645871+0000","last_deep_scrub":"31'57","last_deep_scrub_stamp":"2024-09-17T05:36:40.645871+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.645871+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:45.017498+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":8212,"num_write":40,"num_write_kb":8220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"31'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.704296+0000","last_change":"2024-09-17T05:36:35.704169+0000","last_active":"2024-09-17T05:36:35.704296+0000","last_peered":"2024-09-17T05:36:35.704296+0000","last_clean":"2024-09-17T05:36:35.704296+0000","last_became_active":"2024-09-17T05:27:26.663363+0000","last_became_peered":"2024-09-17T05:27:26.663363+0000","last_unstale":"2024-09-17T05:36:35.704296+0000","last_undegraded":"2024-09-17T05:36:35.704296+0000","last_fullsized":"2024-09-17T05:36:35.704296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'7","last_scrub_stamp":"2024-09-17T05:36:35.701723+0000","last_deep_scrub":"28'7","last_deep_scrub_stamp":"2024-09-17T05:36:35.701723+0000","last_clean_scrub_stamp":"2024-09-17T05:36:35.701723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:41.922299+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"31'5401","reported_seq":5458,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.336170+0000","last_change":"2024-09-17T05:36:50.336170+0000","last_active":"2024-09-17T05:36:50.336170+0000","last_peered":"2024-09-17T05:36:50.336170+0000","last_clean":"2024-09-17T05:36:50.336170+0000","last_became_active":"2024-09-17T05:27:30.673348+0000","last_became_peered":"2024-09-17T05:27:30.673348+0000","last_unstale":"2024-09-17T05:36:50.336170+0000","last_undegraded":"2024-09-17T05:36:50.336170+0000","last_fullsized":"2024-09-17T05:36:50.336170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5401","last_scrub_stamp":"2024-09-17T05:36:50.336047+0000","last_deep_scrub":"31'5401","last_deep_scrub_stamp":"2024-09-17T05:36:50.336047+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.336047+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:23:22.077821+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79473,"num_write_kb":982831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3,6],"acting":[2,5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"31'205","reported_seq":254,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:35.674515+0000","last_change":"2024-09-17T05:36:35.674394+0000","last_active":"2024-09-17T05:36:35.674515+0000","last_peered":"2024-09-17T05:36:35.674515+0000","last_clean":"2024-09-17T05:36:35.674515+0000","last_became_active":"2024-09-17T05:27:25.656815+0000","last_became_peered":"2024-09-17T05:27:25.656815+0000","last_unstale":"2024-09-17T05:36:35.674515+0000","last_undegraded":"2024-09-17T05:36:35.674515+0000","last_fullsized":"2024-09-17T05:36:35.674515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'204","last_scrub_stamp":"2024-09-17T05:36:35.674263+0000","last_deep_scrub":"31'204","last_deep_scrub_stamp":"2024-09-17T05:36:35.674263+0000","last_clean_scrub_stamp":"2024-09-17T05:36:35.674263+0000","objects_scrubbed":2,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:04.566769+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2480742,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":83,"num_write":202,"num_write_kb":2537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.364939+0000","last_change":"2024-09-17T05:36:53.364939+0000","last_active":"2024-09-17T05:36:53.364939+0000","last_peered":"2024-09-17T05:36:53.364939+0000","last_clean":"2024-09-17T05:36:53.364939+0000","last_became_active":"2024-09-17T05:27:26.661243+0000","last_became_peered":"2024-09-17T05:27:26.661243+0000","last_unstale":"2024-09-17T05:36:53.364939+0000","last_undegraded":"2024-09-17T05:36:53.364939+0000","last_fullsized":"2024-09-17T05:36:53.364939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:53.364837+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:53.364837+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.364837+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:28:59.049700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"31'4634","reported_seq":4689,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.469730+0000","last_change":"2024-09-17T05:36:44.469730+0000","last_active":"2024-09-17T05:36:44.469730+0000","last_peered":"2024-09-17T05:36:44.469730+0000","last_clean":"2024-09-17T05:36:44.469730+0000","last_became_active":"2024-09-17T05:27:30.685338+0000","last_became_peered":"2024-09-17T05:27:30.685338+0000","last_unstale":"2024-09-17T05:36:44.469730+0000","last_undegraded":"2024-09-17T05:36:44.469730+0000","last_fullsized":"2024-09-17T05:36:44.469730+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4634","last_scrub_stamp":"2024-09-17T05:36:44.469613+0000","last_deep_scrub":"31'4634","last_deep_scrub_stamp":"2024-09-17T05:36:44.469613+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.469613+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:13.362120+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":68258,"num_write_kb":860103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5,7],"acting":[2,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:37:01.442549+0000","last_change":"2024-09-17T05:37:01.442549+0000","last_active":"2024-09-17T05:37:01.442549+0000","last_peered":"2024-09-17T05:37:01.442549+0000","last_clean":"2024-09-17T05:37:01.442549+0000","last_became_active":"2024-09-17T05:27:26.661299+0000","last_became_peered":"2024-09-17T05:27:26.661299+0000","last_unstale":"2024-09-17T05:37:01.442549+0000","last_undegraded":"2024-09-17T05:37:01.442549+0000","last_fullsized":"2024-09-17T05:37:01.442549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:37:01.442438+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:37:01.442438+0000","last_clean_scrub_stamp":"2024-09-17T05:37:01.442438+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:34.992365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"31'133","reported_seq":184,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.514404+0000","last_change":"2024-09-17T05:36:44.514404+0000","last_active":"2024-09-17T05:36:44.514404+0000","last_peered":"2024-09-17T05:36:44.514404+0000","last_clean":"2024-09-17T05:36:44.514404+0000","last_became_active":"2024-09-17T05:27:25.661905+0000","last_became_peered":"2024-09-17T05:27:25.661905+0000","last_unstale":"2024-09-17T05:36:44.514404+0000","last_undegraded":"2024-09-17T05:36:44.514404+0000","last_fullsized":"2024-09-17T05:36:44.514404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'133","last_scrub_stamp":"2024-09-17T05:36:44.514300+0000","last_deep_scrub":"31'133","last_deep_scrub_stamp":"2024-09-17T05:36:44.514300+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.514300+0000","objects_scrubbed":3,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:14.968108+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4104,"num_write":119,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"31'5681","reported_seq":5731,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:57.441734+0000","last_change":"2024-09-17T05:36:57.441734+0000","last_active":"2024-09-17T05:36:57.441734+0000","last_peered":"2024-09-17T05:36:57.441734+0000","last_clean":"2024-09-17T05:36:57.441734+0000","last_became_active":"2024-09-17T05:27:30.686097+0000","last_became_peered":"2024-09-17T05:27:30.686097+0000","last_unstale":"2024-09-17T05:36:57.441734+0000","last_undegraded":"2024-09-17T05:36:57.441734+0000","last_fullsized":"2024-09-17T05:36:57.441734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5681","last_scrub_stamp":"2024-09-17T05:36:57.441620+0000","last_deep_scrub":"31'5681","last_deep_scrub_stamp":"2024-09-17T05:36:57.441620+0000","last_clean_scrub_stamp":"2024-09-17T05:36:57.441620+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:01.952765+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":83743,"num_write_kb":1023293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,5],"acting":[3,1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.027181+0000","last_change":"2024-09-17T05:36:41.027181+0000","last_active":"2024-09-17T05:36:41.027181+0000","last_peered":"2024-09-17T05:36:41.027181+0000","last_clean":"2024-09-17T05:36:41.027181+0000","last_became_active":"2024-09-17T05:27:26.660994+0000","last_became_peered":"2024-09-17T05:27:26.660994+0000","last_unstale":"2024-09-17T05:36:41.027181+0000","last_undegraded":"2024-09-17T05:36:41.027181+0000","last_fullsized":"2024-09-17T05:36:41.027181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:41.027074+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:41.027074+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.027074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:40.234211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"31'11","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.110113+0000","last_change":"2024-09-17T05:36:45.110113+0000","last_active":"2024-09-17T05:36:45.110113+0000","last_peered":"2024-09-17T05:36:45.110113+0000","last_clean":"2024-09-17T05:36:45.110113+0000","last_became_active":"2024-09-17T05:27:25.655897+0000","last_became_peered":"2024-09-17T05:27:25.655897+0000","last_unstale":"2024-09-17T05:36:45.110113+0000","last_undegraded":"2024-09-17T05:36:45.110113+0000","last_fullsized":"2024-09-17T05:36:45.110113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:45.110011+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:45.110011+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.110011+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:14.795400+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"31'5558","reported_seq":5611,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.618305+0000","last_change":"2024-09-17T05:36:39.618305+0000","last_active":"2024-09-17T05:36:39.618305+0000","last_peered":"2024-09-17T05:36:39.618305+0000","last_clean":"2024-09-17T05:36:39.618305+0000","last_became_active":"2024-09-17T05:27:30.679868+0000","last_became_peered":"2024-09-17T05:27:30.679868+0000","last_unstale":"2024-09-17T05:36:39.618305+0000","last_undegraded":"2024-09-17T05:36:39.618305+0000","last_fullsized":"2024-09-17T05:36:39.618305+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5558","last_scrub_stamp":"2024-09-17T05:36:39.618200+0000","last_deep_scrub":"31'5558","last_deep_scrub_stamp":"2024-09-17T05:36:39.618200+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.618200+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:48.581721+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":81978,"num_write_kb":1031804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6,1],"acting":[0,4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.624406+0000","last_change":"2024-09-17T05:36:50.624406+0000","last_active":"2024-09-17T05:36:50.624406+0000","last_peered":"2024-09-17T05:36:50.624406+0000","last_clean":"2024-09-17T05:36:50.624406+0000","last_became_active":"2024-09-17T05:27:26.655160+0000","last_became_peered":"2024-09-17T05:27:26.655160+0000","last_unstale":"2024-09-17T05:36:50.624406+0000","last_undegraded":"2024-09-17T05:36:50.624406+0000","last_fullsized":"2024-09-17T05:36:50.624406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:50.624312+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:50.624312+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.624312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:11.176834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"31'9","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.661181+0000","last_change":"2024-09-17T05:36:52.661181+0000","last_active":"2024-09-17T05:36:52.661181+0000","last_peered":"2024-09-17T05:36:52.661181+0000","last_clean":"2024-09-17T05:36:52.661181+0000","last_became_active":"2024-09-17T05:27:25.656582+0000","last_became_peered":"2024-09-17T05:27:25.656582+0000","last_unstale":"2024-09-17T05:36:52.661181+0000","last_undegraded":"2024-09-17T05:36:52.661181+0000","last_fullsized":"2024-09-17T05:36:52.661181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:52.661090+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:52.661090+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.661090+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:50.537770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'6547","reported_seq":6596,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.567154+0000","last_change":"2024-09-17T05:36:51.567078+0000","last_active":"2024-09-17T05:36:51.567154+0000","last_peered":"2024-09-17T05:36:51.567154+0000","last_clean":"2024-09-17T05:36:51.567154+0000","last_became_active":"2024-09-17T05:28:07.483263+0000","last_became_peered":"2024-09-17T05:28:07.483263+0000","last_unstale":"2024-09-17T05:36:51.567154+0000","last_undegraded":"2024-09-17T05:36:51.567154+0000","last_fullsized":"2024-09-17T05:36:51.567154+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6547","last_scrub_stamp":"2024-09-17T05:36:51.566959+0000","last_deep_scrub":"31'6547","last_deep_scrub_stamp":"2024-09-17T05:36:51.566959+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.566959+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:24:43.977342+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":96660,"num_write_kb":1196264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2,3],"acting":[0,4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.407845+0000","last_change":"2024-09-17T05:36:41.407845+0000","last_active":"2024-09-17T05:36:41.407845+0000","last_peered":"2024-09-17T05:36:41.407845+0000","last_clean":"2024-09-17T05:36:41.407845+0000","last_became_active":"2024-09-17T05:27:26.655874+0000","last_became_peered":"2024-09-17T05:27:26.655874+0000","last_unstale":"2024-09-17T05:36:41.407845+0000","last_undegraded":"2024-09-17T05:36:41.407845+0000","last_fullsized":"2024-09-17T05:36:41.407845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:41.407755+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:41.407755+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.407755+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:34.958656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"31'17","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.014975+0000","last_change":"2024-09-17T05:36:34.672209+0000","last_active":"2024-09-17T05:36:38.014975+0000","last_peered":"2024-09-17T05:36:38.014975+0000","last_clean":"2024-09-17T05:36:38.014975+0000","last_became_active":"2024-09-17T05:27:25.659960+0000","last_became_peered":"2024-09-17T05:27:25.659960+0000","last_unstale":"2024-09-17T05:36:38.014975+0000","last_undegraded":"2024-09-17T05:36:38.014975+0000","last_fullsized":"2024-09-17T05:36:38.014975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:34.672070+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:34.672070+0000","last_clean_scrub_stamp":"2024-09-17T05:36:34.672070+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:12.516206+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":12,"num_write_kb":4101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"31'4858","reported_seq":4911,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.457817+0000","last_change":"2024-09-17T05:36:45.457817+0000","last_active":"2024-09-17T05:36:45.457817+0000","last_peered":"2024-09-17T05:36:45.457817+0000","last_clean":"2024-09-17T05:36:45.457817+0000","last_became_active":"2024-09-17T05:27:30.677357+0000","last_became_peered":"2024-09-17T05:27:30.677357+0000","last_unstale":"2024-09-17T05:36:45.457817+0000","last_undegraded":"2024-09-17T05:36:45.457817+0000","last_fullsized":"2024-09-17T05:36:45.457817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4858","last_scrub_stamp":"2024-09-17T05:36:45.457699+0000","last_deep_scrub":"31'4858","last_deep_scrub_stamp":"2024-09-17T05:36:45.457699+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.457699+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:18.190686+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":71425,"num_write_kb":889518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,7],"acting":[2,1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'19","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.585112+0000","last_change":"2024-09-17T05:36:42.585112+0000","last_active":"2024-09-17T05:36:42.585112+0000","last_peered":"2024-09-17T05:36:42.585112+0000","last_clean":"2024-09-17T05:36:42.585112+0000","last_became_active":"2024-09-17T05:27:26.655350+0000","last_became_peered":"2024-09-17T05:27:26.655350+0000","last_unstale":"2024-09-17T05:36:42.585112+0000","last_undegraded":"2024-09-17T05:36:42.585112+0000","last_fullsized":"2024-09-17T05:36:42.585112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:42.585017+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:42.585017+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.585017+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:15.318004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"31'10","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.113998+0000","last_change":"2024-09-17T05:36:37.501894+0000","last_active":"2024-09-17T05:36:38.113998+0000","last_peered":"2024-09-17T05:36:38.113998+0000","last_clean":"2024-09-17T05:36:38.113998+0000","last_became_active":"2024-09-17T05:27:25.655742+0000","last_became_peered":"2024-09-17T05:27:25.655742+0000","last_unstale":"2024-09-17T05:36:38.113998+0000","last_undegraded":"2024-09-17T05:36:38.113998+0000","last_fullsized":"2024-09-17T05:36:38.113998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:37.501768+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:37.501768+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.501768+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:58.829230+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"31'5547","reported_seq":5604,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.416286+0000","last_change":"2024-09-17T05:36:46.416286+0000","last_active":"2024-09-17T05:36:46.416286+0000","last_peered":"2024-09-17T05:36:46.416286+0000","last_clean":"2024-09-17T05:36:46.416286+0000","last_became_active":"2024-09-17T05:27:30.671712+0000","last_became_peered":"2024-09-17T05:27:30.671712+0000","last_unstale":"2024-09-17T05:36:46.416286+0000","last_undegraded":"2024-09-17T05:36:46.416286+0000","last_fullsized":"2024-09-17T05:36:46.416286+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5547","last_scrub_stamp":"2024-09-17T05:36:46.416107+0000","last_deep_scrub":"31'5547","last_deep_scrub_stamp":"2024-09-17T05:36:46.416107+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.416107+0000","objects_scrubbed":0,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:12.759610+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":81766,"num_write_kb":1039133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,1],"acting":[2,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.625891+0000","last_change":"2024-09-17T05:36:38.625891+0000","last_active":"2024-09-17T05:36:38.625891+0000","last_peered":"2024-09-17T05:36:38.625891+0000","last_clean":"2024-09-17T05:36:38.625891+0000","last_became_active":"2024-09-17T05:27:26.653566+0000","last_became_peered":"2024-09-17T05:27:26.653566+0000","last_unstale":"2024-09-17T05:36:38.625891+0000","last_undegraded":"2024-09-17T05:36:38.625891+0000","last_fullsized":"2024-09-17T05:36:38.625891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:38.625752+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:38.625752+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.625752+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:05.595811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"31'225","reported_seq":277,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.602460+0000","last_change":"2024-09-17T05:36:40.602460+0000","last_active":"2024-09-17T05:36:40.602460+0000","last_peered":"2024-09-17T05:36:40.602460+0000","last_clean":"2024-09-17T05:36:40.602460+0000","last_became_active":"2024-09-17T05:27:25.653667+0000","last_became_peered":"2024-09-17T05:27:25.653667+0000","last_unstale":"2024-09-17T05:36:40.602460+0000","last_undegraded":"2024-09-17T05:36:40.602460+0000","last_fullsized":"2024-09-17T05:36:40.602460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'225","last_scrub_stamp":"2024-09-17T05:36:40.602363+0000","last_deep_scrub":"31'225","last_deep_scrub_stamp":"2024-09-17T05:36:40.602363+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.602363+0000","objects_scrubbed":2,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:10.287752+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":21,"num_write":446,"num_write_kb":181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"31'6023","reported_seq":6082,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.684015+0000","last_change":"2024-09-17T05:36:51.684015+0000","last_active":"2024-09-17T05:36:51.684015+0000","last_peered":"2024-09-17T05:36:51.684015+0000","last_clean":"2024-09-17T05:36:51.684015+0000","last_became_active":"2024-09-17T05:27:30.672355+0000","last_became_peered":"2024-09-17T05:27:30.672355+0000","last_unstale":"2024-09-17T05:36:51.684015+0000","last_undegraded":"2024-09-17T05:36:51.684015+0000","last_fullsized":"2024-09-17T05:36:51.684015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6023","last_scrub_stamp":"2024-09-17T05:36:51.683904+0000","last_deep_scrub":"31'6023","last_deep_scrub_stamp":"2024-09-17T05:36:51.683904+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.683904+0000","objects_scrubbed":0,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:05:39.444905+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":88895,"num_write_kb":1123429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6,3],"acting":[4,1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.032926+0000","last_change":"2024-09-17T05:36:41.032926+0000","last_active":"2024-09-17T05:36:41.032926+0000","last_peered":"2024-09-17T05:36:41.032926+0000","last_clean":"2024-09-17T05:36:41.032926+0000","last_became_active":"2024-09-17T05:27:26.657976+0000","last_became_peered":"2024-09-17T05:27:26.657976+0000","last_unstale":"2024-09-17T05:36:41.032926+0000","last_undegraded":"2024-09-17T05:36:41.032926+0000","last_fullsized":"2024-09-17T05:36:41.032926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:41.032832+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:41.032832+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.032832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:57.520423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"31'6","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.448891+0000","last_change":"2024-09-17T05:36:51.448891+0000","last_active":"2024-09-17T05:36:51.448891+0000","last_peered":"2024-09-17T05:36:51.448891+0000","last_clean":"2024-09-17T05:36:51.448891+0000","last_became_active":"2024-09-17T05:27:25.645028+0000","last_became_peered":"2024-09-17T05:27:25.645028+0000","last_unstale":"2024-09-17T05:36:51.448891+0000","last_undegraded":"2024-09-17T05:36:51.448891+0000","last_fullsized":"2024-09-17T05:36:51.448891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-09-17T05:36:51.448792+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-09-17T05:36:51.448792+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.448792+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:32.191573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'5078","reported_seq":5128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.279852+0000","last_change":"2024-09-17T05:36:50.279852+0000","last_active":"2024-09-17T05:36:50.279852+0000","last_peered":"2024-09-17T05:36:50.279852+0000","last_clean":"2024-09-17T05:36:50.279852+0000","last_became_active":"2024-09-17T05:27:30.677293+0000","last_became_peered":"2024-09-17T05:27:30.677293+0000","last_unstale":"2024-09-17T05:36:50.279852+0000","last_undegraded":"2024-09-17T05:36:50.279852+0000","last_fullsized":"2024-09-17T05:36:50.279852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5078","last_scrub_stamp":"2024-09-17T05:36:50.279730+0000","last_deep_scrub":"31'5078","last_deep_scrub_stamp":"2024-09-17T05:36:50.279730+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.279730+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:59.243251+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":75020,"num_write_kb":939196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2,7],"acting":[6,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.323231+0000","last_change":"2024-09-17T05:36:38.323231+0000","last_active":"2024-09-17T05:36:38.323231+0000","last_peered":"2024-09-17T05:36:38.323231+0000","last_clean":"2024-09-17T05:36:38.323231+0000","last_became_active":"2024-09-17T05:27:26.654898+0000","last_became_peered":"2024-09-17T05:27:26.654898+0000","last_unstale":"2024-09-17T05:36:38.323231+0000","last_undegraded":"2024-09-17T05:36:38.323231+0000","last_fullsized":"2024-09-17T05:36:38.323231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:38.323156+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:38.323156+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.323156+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:31.500391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"31'5","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.220388+0000","last_change":"2024-09-17T05:36:44.220388+0000","last_active":"2024-09-17T05:36:44.220388+0000","last_peered":"2024-09-17T05:36:44.220388+0000","last_clean":"2024-09-17T05:36:44.220388+0000","last_became_active":"2024-09-17T05:27:25.652231+0000","last_became_peered":"2024-09-17T05:27:25.652231+0000","last_unstale":"2024-09-17T05:36:44.220388+0000","last_undegraded":"2024-09-17T05:36:44.220388+0000","last_fullsized":"2024-09-17T05:36:44.220388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-09-17T05:36:44.220283+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-09-17T05:36:44.220283+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.220283+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:09.189959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"31'6224","reported_seq":6280,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.336045+0000","last_change":"2024-09-17T05:36:52.336045+0000","last_active":"2024-09-17T05:36:52.336045+0000","last_peered":"2024-09-17T05:36:52.336045+0000","last_clean":"2024-09-17T05:36:52.336045+0000","last_became_active":"2024-09-17T05:27:30.687174+0000","last_became_peered":"2024-09-17T05:27:30.687174+0000","last_unstale":"2024-09-17T05:36:52.336045+0000","last_undegraded":"2024-09-17T05:36:52.336045+0000","last_fullsized":"2024-09-17T05:36:52.336045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6224","last_scrub_stamp":"2024-09-17T05:36:52.335891+0000","last_deep_scrub":"31'6224","last_deep_scrub_stamp":"2024-09-17T05:36:52.335891+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.335891+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:40:59.226496+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":92055,"num_write_kb":1145505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5,7],"acting":[6,3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.685814+0000","last_change":"2024-09-17T05:36:47.685814+0000","last_active":"2024-09-17T05:36:47.685814+0000","last_peered":"2024-09-17T05:36:47.685814+0000","last_clean":"2024-09-17T05:36:47.685814+0000","last_became_active":"2024-09-17T05:27:26.658191+0000","last_became_peered":"2024-09-17T05:27:26.658191+0000","last_unstale":"2024-09-17T05:36:47.685814+0000","last_undegraded":"2024-09-17T05:36:47.685814+0000","last_fullsized":"2024-09-17T05:36:47.685814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:47.685701+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:47.685701+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.685701+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:08.285051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"31'47","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:58.395581+0000","last_change":"2024-09-17T05:36:58.395581+0000","last_active":"2024-09-17T05:36:58.395581+0000","last_peered":"2024-09-17T05:36:58.395581+0000","last_clean":"2024-09-17T05:36:58.395581+0000","last_became_active":"2024-09-17T05:27:25.652865+0000","last_became_peered":"2024-09-17T05:27:25.652865+0000","last_unstale":"2024-09-17T05:36:58.395581+0000","last_undegraded":"2024-09-17T05:36:58.395581+0000","last_fullsized":"2024-09-17T05:36:58.395581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-09-17T05:36:58.395483+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-09-17T05:36:58.395483+0000","last_clean_scrub_stamp":"2024-09-17T05:36:58.395483+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:41:20.810082+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":35,"num_write_kb":8210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'5360","reported_seq":5418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.256733+0000","last_change":"2024-09-17T05:36:49.256733+0000","last_active":"2024-09-17T05:36:49.256733+0000","last_peered":"2024-09-17T05:36:49.256733+0000","last_clean":"2024-09-17T05:36:49.256733+0000","last_became_active":"2024-09-17T05:27:30.685931+0000","last_became_peered":"2024-09-17T05:27:30.685931+0000","last_unstale":"2024-09-17T05:36:49.256733+0000","last_undegraded":"2024-09-17T05:36:49.256733+0000","last_fullsized":"2024-09-17T05:36:49.256733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5360","last_scrub_stamp":"2024-09-17T05:36:49.256614+0000","last_deep_scrub":"31'5360","last_deep_scrub_stamp":"2024-09-17T05:36:49.256614+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.256614+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:19:07.048092+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79033,"num_write_kb":985615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0,3],"acting":[6,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.248530+0000","last_change":"2024-09-17T05:36:41.248530+0000","last_active":"2024-09-17T05:36:41.248530+0000","last_peered":"2024-09-17T05:36:41.248530+0000","last_clean":"2024-09-17T05:36:41.248530+0000","last_became_active":"2024-09-17T05:27:26.656268+0000","last_became_peered":"2024-09-17T05:27:26.656268+0000","last_unstale":"2024-09-17T05:36:41.248530+0000","last_undegraded":"2024-09-17T05:36:41.248530+0000","last_fullsized":"2024-09-17T05:36:41.248530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:41.248425+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:41.248425+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.248425+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:20.776332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"31'33","reported_seq":102,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.066242+0000","last_change":"2024-09-17T05:36:40.066242+0000","last_active":"2024-09-17T05:36:40.066242+0000","last_peered":"2024-09-17T05:36:40.066242+0000","last_clean":"2024-09-17T05:36:40.066242+0000","last_became_active":"2024-09-17T05:27:25.655767+0000","last_became_peered":"2024-09-17T05:27:25.655767+0000","last_unstale":"2024-09-17T05:36:40.066242+0000","last_undegraded":"2024-09-17T05:36:40.066242+0000","last_fullsized":"2024-09-17T05:36:40.066242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'33","last_scrub_stamp":"2024-09-17T05:36:40.066094+0000","last_deep_scrub":"31'33","last_deep_scrub_stamp":"2024-09-17T05:36:40.066094+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.066094+0000","objects_scrubbed":3,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:02.733563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94359,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":116,"num_write":27,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"31'5929","reported_seq":5988,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.307148+0000","last_change":"2024-09-17T05:36:53.307148+0000","last_active":"2024-09-17T05:36:53.307148+0000","last_peered":"2024-09-17T05:36:53.307148+0000","last_clean":"2024-09-17T05:36:53.307148+0000","last_became_active":"2024-09-17T05:27:30.677686+0000","last_became_peered":"2024-09-17T05:27:30.677686+0000","last_unstale":"2024-09-17T05:36:53.307148+0000","last_undegraded":"2024-09-17T05:36:53.307148+0000","last_fullsized":"2024-09-17T05:36:53.307148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5929","last_scrub_stamp":"2024-09-17T05:36:53.307008+0000","last_deep_scrub":"31'5929","last_deep_scrub_stamp":"2024-09-17T05:36:53.307008+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.307008+0000","objects_scrubbed":0,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:20.214495+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":0,"num_write":87395,"num_write_kb":1093347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,7],"acting":[6,0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"31'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.078516+0000","last_change":"2024-09-17T05:36:42.078516+0000","last_active":"2024-09-17T05:36:42.078516+0000","last_peered":"2024-09-17T05:36:42.078516+0000","last_clean":"2024-09-17T05:36:42.078516+0000","last_became_active":"2024-09-17T05:27:26.653350+0000","last_became_peered":"2024-09-17T05:27:26.653350+0000","last_unstale":"2024-09-17T05:36:42.078516+0000","last_undegraded":"2024-09-17T05:36:42.078516+0000","last_fullsized":"2024-09-17T05:36:42.078516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:42.078430+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:42.078430+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.078430+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:48.066042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"31'98","reported_seq":151,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.485406+0000","last_change":"2024-09-17T05:36:46.485406+0000","last_active":"2024-09-17T05:36:46.485406+0000","last_peered":"2024-09-17T05:36:46.485406+0000","last_clean":"2024-09-17T05:36:46.485406+0000","last_became_active":"2024-09-17T05:27:25.658309+0000","last_became_peered":"2024-09-17T05:27:25.658309+0000","last_unstale":"2024-09-17T05:36:46.485406+0000","last_undegraded":"2024-09-17T05:36:46.485406+0000","last_fullsized":"2024-09-17T05:36:46.485406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'98","last_scrub_stamp":"2024-09-17T05:36:46.485320+0000","last_deep_scrub":"31'98","last_deep_scrub_stamp":"2024-09-17T05:36:46.485320+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.485320+0000","objects_scrubbed":3,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:27.174887+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8204,"num_write":88,"num_write_kb":8238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"31'4825","reported_seq":4875,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:55.355859+0000","last_change":"2024-09-17T05:36:55.355859+0000","last_active":"2024-09-17T05:36:55.355859+0000","last_peered":"2024-09-17T05:36:55.355859+0000","last_clean":"2024-09-17T05:36:55.355859+0000","last_became_active":"2024-09-17T05:27:30.688925+0000","last_became_peered":"2024-09-17T05:27:30.688925+0000","last_unstale":"2024-09-17T05:36:55.355859+0000","last_undegraded":"2024-09-17T05:36:55.355859+0000","last_fullsized":"2024-09-17T05:36:55.355859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4825","last_scrub_stamp":"2024-09-17T05:36:55.355734+0000","last_deep_scrub":"31'4825","last_deep_scrub_stamp":"2024-09-17T05:36:55.355734+0000","last_clean_scrub_stamp":"2024-09-17T05:36:55.355734+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:58.879206+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":0,"num_write":71019,"num_write_kb":889582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6,1],"acting":[5,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"31'8","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.701641+0000","last_change":"2024-09-17T05:36:45.701641+0000","last_active":"2024-09-17T05:36:45.701641+0000","last_peered":"2024-09-17T05:36:45.701641+0000","last_clean":"2024-09-17T05:36:45.701641+0000","last_became_active":"2024-09-17T05:27:26.652639+0000","last_became_peered":"2024-09-17T05:27:26.652639+0000","last_unstale":"2024-09-17T05:36:45.701641+0000","last_undegraded":"2024-09-17T05:36:45.701641+0000","last_fullsized":"2024-09-17T05:36:45.701641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:45.701541+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:45.701541+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.701541+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:17:54.665061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'9","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.432601+0000","last_change":"2024-09-17T05:36:42.432601+0000","last_active":"2024-09-17T05:36:42.432601+0000","last_peered":"2024-09-17T05:36:42.432601+0000","last_clean":"2024-09-17T05:36:42.432601+0000","last_became_active":"2024-09-17T05:27:25.661889+0000","last_became_peered":"2024-09-17T05:27:25.661889+0000","last_unstale":"2024-09-17T05:36:42.432601+0000","last_undegraded":"2024-09-17T05:36:42.432601+0000","last_fullsized":"2024-09-17T05:36:42.432601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:42.432508+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:42.432508+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.432508+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:00.017778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":11,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"31'74","reported_seq":145,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:36.709521+0000","last_change":"2024-09-17T05:36:36.709521+0000","last_active":"2024-09-17T05:36:36.709521+0000","last_peered":"2024-09-17T05:36:36.709521+0000","last_clean":"2024-09-17T05:36:36.709521+0000","last_became_active":"2024-09-17T05:27:15.487251+0000","last_became_peered":"2024-09-17T05:27:15.487251+0000","last_unstale":"2024-09-17T05:36:36.709521+0000","last_undegraded":"2024-09-17T05:36:36.709521+0000","last_fullsized":"2024-09-17T05:36:36.709521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'74","last_scrub_stamp":"2024-09-17T05:36:36.703771+0000","last_deep_scrub":"31'74","last_deep_scrub_stamp":"2024-09-17T05:36:36.703771+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.703771+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:32.886636+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'5418","reported_seq":5475,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:55.407859+0000","last_change":"2024-09-17T05:36:55.407859+0000","last_active":"2024-09-17T05:36:55.407859+0000","last_peered":"2024-09-17T05:36:55.407859+0000","last_clean":"2024-09-17T05:36:55.407859+0000","last_became_active":"2024-09-17T05:27:30.673512+0000","last_became_peered":"2024-09-17T05:27:30.673512+0000","last_unstale":"2024-09-17T05:36:55.407859+0000","last_undegraded":"2024-09-17T05:36:55.407859+0000","last_fullsized":"2024-09-17T05:36:55.407859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5418","last_scrub_stamp":"2024-09-17T05:36:55.407708+0000","last_deep_scrub":"31'5418","last_deep_scrub_stamp":"2024-09-17T05:36:55.407708+0000","last_clean_scrub_stamp":"2024-09-17T05:36:55.407708+0000","objects_scrubbed":0,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:01.932731+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":80049,"num_write_kb":1015697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2,0],"acting":[3,1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.403680+0000","last_change":"2024-09-17T05:36:52.403680+0000","last_active":"2024-09-17T05:36:52.403680+0000","last_peered":"2024-09-17T05:36:52.403680+0000","last_clean":"2024-09-17T05:36:52.403680+0000","last_became_active":"2024-09-17T05:27:26.662270+0000","last_became_peered":"2024-09-17T05:27:26.662270+0000","last_unstale":"2024-09-17T05:36:52.403680+0000","last_undegraded":"2024-09-17T05:36:52.403680+0000","last_fullsized":"2024-09-17T05:36:52.403680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:52.403589+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:52.403589+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.403589+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:05.144325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.136774+0000","last_change":"2024-09-17T05:36:44.136774+0000","last_active":"2024-09-17T05:36:44.136774+0000","last_peered":"2024-09-17T05:36:44.136774+0000","last_clean":"2024-09-17T05:36:44.136774+0000","last_became_active":"2024-09-17T05:27:25.660802+0000","last_became_peered":"2024-09-17T05:27:25.660802+0000","last_unstale":"2024-09-17T05:36:44.136774+0000","last_undegraded":"2024-09-17T05:36:44.136774+0000","last_fullsized":"2024-09-17T05:36:44.136774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:44.136684+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:44.136684+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.136684+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:19.125250+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"31'5576","reported_seq":5632,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.433064+0000","last_change":"2024-09-17T05:36:43.433064+0000","last_active":"2024-09-17T05:36:43.433064+0000","last_peered":"2024-09-17T05:36:43.433064+0000","last_clean":"2024-09-17T05:36:43.433064+0000","last_became_active":"2024-09-17T05:27:30.677370+0000","last_became_peered":"2024-09-17T05:27:30.677370+0000","last_unstale":"2024-09-17T05:36:43.433064+0000","last_undegraded":"2024-09-17T05:36:43.433064+0000","last_fullsized":"2024-09-17T05:36:43.433064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5576","last_scrub_stamp":"2024-09-17T05:36:43.432953+0000","last_deep_scrub":"31'5576","last_deep_scrub_stamp":"2024-09-17T05:36:43.432953+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.432953+0000","objects_scrubbed":0,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:21.277089+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82316,"num_write_kb":1025442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7,4],"acting":[2,3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'14","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.139906+0000","last_change":"2024-09-17T05:36:36.597250+0000","last_active":"2024-09-17T05:36:38.139906+0000","last_peered":"2024-09-17T05:36:38.139906+0000","last_clean":"2024-09-17T05:36:38.139906+0000","last_became_active":"2024-09-17T05:27:26.658825+0000","last_became_peered":"2024-09-17T05:27:26.658825+0000","last_unstale":"2024-09-17T05:36:38.139906+0000","last_undegraded":"2024-09-17T05:36:38.139906+0000","last_fullsized":"2024-09-17T05:36:38.139906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:36.597064+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:36.597064+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.597064+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:09.330050+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"31'23","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.253255+0000","last_change":"2024-09-17T05:36:40.253255+0000","last_active":"2024-09-17T05:36:40.253255+0000","last_peered":"2024-09-17T05:36:40.253255+0000","last_clean":"2024-09-17T05:36:40.253255+0000","last_became_active":"2024-09-17T05:27:25.652850+0000","last_became_peered":"2024-09-17T05:27:25.652850+0000","last_unstale":"2024-09-17T05:36:40.253255+0000","last_undegraded":"2024-09-17T05:36:40.253255+0000","last_fullsized":"2024-09-17T05:36:40.253255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-09-17T05:36:40.253130+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-09-17T05:36:40.253130+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.253130+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:45:45.419407+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":4097,"num_write":19,"num_write_kb":4105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"31'5086","reported_seq":5143,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:56.376470+0000","last_change":"2024-09-17T05:36:56.376470+0000","last_active":"2024-09-17T05:36:56.376470+0000","last_peered":"2024-09-17T05:36:56.376470+0000","last_clean":"2024-09-17T05:36:56.376470+0000","last_became_active":"2024-09-17T05:27:30.679554+0000","last_became_peered":"2024-09-17T05:27:30.679554+0000","last_unstale":"2024-09-17T05:36:56.376470+0000","last_undegraded":"2024-09-17T05:36:56.376470+0000","last_fullsized":"2024-09-17T05:36:56.376470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5086","last_scrub_stamp":"2024-09-17T05:36:56.376313+0000","last_deep_scrub":"31'5086","last_deep_scrub_stamp":"2024-09-17T05:36:56.376313+0000","last_clean_scrub_stamp":"2024-09-17T05:36:56.376313+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:31.672873+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":74966,"num_write_kb":933970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1,3],"acting":[5,4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.548955+0000","last_change":"2024-09-17T05:36:50.548955+0000","last_active":"2024-09-17T05:36:50.548955+0000","last_peered":"2024-09-17T05:36:50.548955+0000","last_clean":"2024-09-17T05:36:50.548955+0000","last_became_active":"2024-09-17T05:27:26.662652+0000","last_became_peered":"2024-09-17T05:27:26.662652+0000","last_unstale":"2024-09-17T05:36:50.548955+0000","last_undegraded":"2024-09-17T05:36:50.548955+0000","last_fullsized":"2024-09-17T05:36:50.548955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:50.548871+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:50.548871+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.548871+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:28.306780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"31'32","reported_seq":83,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.618965+0000","last_change":"2024-09-17T05:36:41.618965+0000","last_active":"2024-09-17T05:36:41.618965+0000","last_peered":"2024-09-17T05:36:41.618965+0000","last_clean":"2024-09-17T05:36:41.618965+0000","last_became_active":"2024-09-17T05:27:25.650986+0000","last_became_peered":"2024-09-17T05:27:25.650986+0000","last_unstale":"2024-09-17T05:36:41.618965+0000","last_undegraded":"2024-09-17T05:36:41.618965+0000","last_fullsized":"2024-09-17T05:36:41.618965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'32","last_scrub_stamp":"2024-09-17T05:36:41.618873+0000","last_deep_scrub":"31'32","last_deep_scrub_stamp":"2024-09-17T05:36:41.618873+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.618873+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:10.511379+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4102,"num_write":26,"num_write_kb":4108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"31'5181","reported_seq":5234,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.491487+0000","last_change":"2024-09-17T05:36:50.491487+0000","last_active":"2024-09-17T05:36:50.491487+0000","last_peered":"2024-09-17T05:36:50.491487+0000","last_clean":"2024-09-17T05:36:50.491487+0000","last_became_active":"2024-09-17T05:27:30.680097+0000","last_became_peered":"2024-09-17T05:27:30.680097+0000","last_unstale":"2024-09-17T05:36:50.491487+0000","last_undegraded":"2024-09-17T05:36:50.491487+0000","last_fullsized":"2024-09-17T05:36:50.491487+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5181","last_scrub_stamp":"2024-09-17T05:36:50.491356+0000","last_deep_scrub":"31'5181","last_deep_scrub_stamp":"2024-09-17T05:36:50.491356+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.491356+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:04.357783+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76474,"num_write_kb":955028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"31'7","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:37.927749+0000","last_change":"2024-09-17T05:36:37.310355+0000","last_active":"2024-09-17T05:36:37.927749+0000","last_peered":"2024-09-17T05:36:37.927749+0000","last_clean":"2024-09-17T05:36:37.927749+0000","last_became_active":"2024-09-17T05:27:26.661021+0000","last_became_peered":"2024-09-17T05:27:26.661021+0000","last_unstale":"2024-09-17T05:36:37.927749+0000","last_undegraded":"2024-09-17T05:36:37.927749+0000","last_fullsized":"2024-09-17T05:36:37.927749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-09-17T05:36:37.309821+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-09-17T05:36:37.309821+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.309821+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:57.303315+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"31'13","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.598972+0000","last_change":"2024-09-17T05:36:49.598972+0000","last_active":"2024-09-17T05:36:49.598972+0000","last_peered":"2024-09-17T05:36:49.598972+0000","last_clean":"2024-09-17T05:36:49.598972+0000","last_became_active":"2024-09-17T05:27:25.657306+0000","last_became_peered":"2024-09-17T05:27:25.657306+0000","last_unstale":"2024-09-17T05:36:49.598972+0000","last_undegraded":"2024-09-17T05:36:49.598972+0000","last_fullsized":"2024-09-17T05:36:49.598972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:49.598830+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:49.598830+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.598830+0000","objects_scrubbed":2,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:32.336296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"31'5199","reported_seq":5252,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.765591+0000","last_change":"2024-09-17T05:36:40.765591+0000","last_active":"2024-09-17T05:36:40.765591+0000","last_peered":"2024-09-17T05:36:40.765591+0000","last_clean":"2024-09-17T05:36:40.765591+0000","last_became_active":"2024-09-17T05:27:30.678342+0000","last_became_peered":"2024-09-17T05:27:30.678342+0000","last_unstale":"2024-09-17T05:36:40.765591+0000","last_undegraded":"2024-09-17T05:36:40.765591+0000","last_fullsized":"2024-09-17T05:36:40.765591+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5199","last_scrub_stamp":"2024-09-17T05:36:40.765481+0000","last_deep_scrub":"31'5199","last_deep_scrub_stamp":"2024-09-17T05:36:40.765481+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.765481+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:03.432950+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":76687,"num_write_kb":970760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,2],"acting":[1,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.453257+0000","last_change":"2024-09-17T05:36:43.453147+0000","last_active":"2024-09-17T05:36:43.453257+0000","last_peered":"2024-09-17T05:36:43.453257+0000","last_clean":"2024-09-17T05:36:43.453257+0000","last_became_active":"2024-09-17T05:27:26.656202+0000","last_became_peered":"2024-09-17T05:27:26.656202+0000","last_unstale":"2024-09-17T05:36:43.453257+0000","last_undegraded":"2024-09-17T05:36:43.453257+0000","last_fullsized":"2024-09-17T05:36:43.453257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:43.453017+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:43.453017+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.453017+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:42:46.999337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"31'31","reported_seq":94,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.662904+0000","last_change":"2024-09-17T05:36:49.662904+0000","last_active":"2024-09-17T05:36:49.662904+0000","last_peered":"2024-09-17T05:36:49.662904+0000","last_clean":"2024-09-17T05:36:49.662904+0000","last_became_active":"2024-09-17T05:27:25.653596+0000","last_became_peered":"2024-09-17T05:27:25.653596+0000","last_unstale":"2024-09-17T05:36:49.662904+0000","last_undegraded":"2024-09-17T05:36:49.662904+0000","last_fullsized":"2024-09-17T05:36:49.662904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'31","last_scrub_stamp":"2024-09-17T05:36:49.662789+0000","last_deep_scrub":"31'31","last_deep_scrub_stamp":"2024-09-17T05:36:49.662789+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.662789+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:14.581798+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":40,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"31'5828","reported_seq":5888,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:57.411133+0000","last_change":"2024-09-17T05:36:57.411026+0000","last_active":"2024-09-17T05:36:57.411133+0000","last_peered":"2024-09-17T05:36:57.411133+0000","last_clean":"2024-09-17T05:36:57.411133+0000","last_became_active":"2024-09-17T05:27:30.688563+0000","last_became_peered":"2024-09-17T05:27:30.688563+0000","last_unstale":"2024-09-17T05:36:57.411133+0000","last_undegraded":"2024-09-17T05:36:57.411133+0000","last_fullsized":"2024-09-17T05:36:57.411133+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5828","last_scrub_stamp":"2024-09-17T05:36:57.410905+0000","last_deep_scrub":"31'5828","last_deep_scrub_stamp":"2024-09-17T05:36:57.410905+0000","last_clean_scrub_stamp":"2024-09-17T05:36:57.410905+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:06.756356+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":85918,"num_write_kb":1086559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1,6],"acting":[5,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.173409+0000","last_change":"2024-09-17T05:36:37.673724+0000","last_active":"2024-09-17T05:36:38.173409+0000","last_peered":"2024-09-17T05:36:38.173409+0000","last_clean":"2024-09-17T05:36:38.173409+0000","last_became_active":"2024-09-17T05:27:26.657609+0000","last_became_peered":"2024-09-17T05:27:26.657609+0000","last_unstale":"2024-09-17T05:36:38.173409+0000","last_undegraded":"2024-09-17T05:36:38.173409+0000","last_fullsized":"2024-09-17T05:36:38.173409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:37.673615+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:37.673615+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.673615+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:25.551966+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"31'19","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.712843+0000","last_change":"2024-09-17T05:36:43.712843+0000","last_active":"2024-09-17T05:36:43.712843+0000","last_peered":"2024-09-17T05:36:43.712843+0000","last_clean":"2024-09-17T05:36:43.712843+0000","last_became_active":"2024-09-17T05:27:25.657168+0000","last_became_peered":"2024-09-17T05:27:25.657168+0000","last_unstale":"2024-09-17T05:36:43.712843+0000","last_undegraded":"2024-09-17T05:36:43.712843+0000","last_fullsized":"2024-09-17T05:36:43.712843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:43.712680+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:43.712680+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.712680+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:08.631689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1689,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":14,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"31'5926","reported_seq":5983,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:58.384257+0000","last_change":"2024-09-17T05:36:58.384176+0000","last_active":"2024-09-17T05:36:58.384257+0000","last_peered":"2024-09-17T05:36:58.384257+0000","last_clean":"2024-09-17T05:36:58.384257+0000","last_became_active":"2024-09-17T05:27:30.685494+0000","last_became_peered":"2024-09-17T05:27:30.685494+0000","last_unstale":"2024-09-17T05:36:58.384257+0000","last_undegraded":"2024-09-17T05:36:58.384257+0000","last_fullsized":"2024-09-17T05:36:58.384257+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5926","last_scrub_stamp":"2024-09-17T05:36:58.384071+0000","last_deep_scrub":"31'5926","last_deep_scrub_stamp":"2024-09-17T05:36:58.384071+0000","last_clean_scrub_stamp":"2024-09-17T05:36:58.384071+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:57.877406+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87434,"num_write_kb":1070021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6,4],"acting":[5,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"31'14","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.496662+0000","last_change":"2024-09-17T05:36:45.496592+0000","last_active":"2024-09-17T05:36:45.496662+0000","last_peered":"2024-09-17T05:36:45.496662+0000","last_clean":"2024-09-17T05:36:45.496662+0000","last_became_active":"2024-09-17T05:27:26.665334+0000","last_became_peered":"2024-09-17T05:27:26.665334+0000","last_unstale":"2024-09-17T05:36:45.496662+0000","last_undegraded":"2024-09-17T05:36:45.496662+0000","last_fullsized":"2024-09-17T05:36:45.496662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:45.496501+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:45.496501+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.496501+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:59.345367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"31'11","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:59.392208+0000","last_change":"2024-09-17T05:36:59.392208+0000","last_active":"2024-09-17T05:36:59.392208+0000","last_peered":"2024-09-17T05:36:59.392208+0000","last_clean":"2024-09-17T05:36:59.392208+0000","last_became_active":"2024-09-17T05:27:25.654077+0000","last_became_peered":"2024-09-17T05:27:25.654077+0000","last_unstale":"2024-09-17T05:36:59.392208+0000","last_undegraded":"2024-09-17T05:36:59.392208+0000","last_fullsized":"2024-09-17T05:36:59.392208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:59.392115+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:59.392115+0000","last_clean_scrub_stamp":"2024-09-17T05:36:59.392115+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:35:54.627191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"31'5945","reported_seq":6002,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.314752+0000","last_change":"2024-09-17T05:36:51.314677+0000","last_active":"2024-09-17T05:36:51.314752+0000","last_peered":"2024-09-17T05:36:51.314752+0000","last_clean":"2024-09-17T05:36:51.314752+0000","last_became_active":"2024-09-17T05:27:30.687109+0000","last_became_peered":"2024-09-17T05:27:30.687109+0000","last_unstale":"2024-09-17T05:36:51.314752+0000","last_undegraded":"2024-09-17T05:36:51.314752+0000","last_fullsized":"2024-09-17T05:36:51.314752+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5945","last_scrub_stamp":"2024-09-17T05:36:51.314565+0000","last_deep_scrub":"31'5945","last_deep_scrub_stamp":"2024-09-17T05:36:51.314565+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.314565+0000","objects_scrubbed":0,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:55.686191+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":0,"num_write":87608,"num_write_kb":1085839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0,7],"acting":[6,4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.018029+0000","last_change":"2024-09-17T05:36:40.017952+0000","last_active":"2024-09-17T05:36:40.018029+0000","last_peered":"2024-09-17T05:36:40.018029+0000","last_clean":"2024-09-17T05:36:40.018029+0000","last_became_active":"2024-09-17T05:27:26.661404+0000","last_became_peered":"2024-09-17T05:27:26.661404+0000","last_unstale":"2024-09-17T05:36:40.018029+0000","last_undegraded":"2024-09-17T05:36:40.018029+0000","last_fullsized":"2024-09-17T05:36:40.018029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:40.017855+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:40.017855+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.017855+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:20.964313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"31'98","reported_seq":154,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.067437+0000","last_change":"2024-09-17T05:36:47.067348+0000","last_active":"2024-09-17T05:36:47.067437+0000","last_peered":"2024-09-17T05:36:47.067437+0000","last_clean":"2024-09-17T05:36:47.067437+0000","last_became_active":"2024-09-17T05:27:25.660551+0000","last_became_peered":"2024-09-17T05:27:25.660551+0000","last_unstale":"2024-09-17T05:36:47.067437+0000","last_undegraded":"2024-09-17T05:36:47.067437+0000","last_fullsized":"2024-09-17T05:36:47.067437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'98","last_scrub_stamp":"2024-09-17T05:36:47.067218+0000","last_deep_scrub":"31'98","last_deep_scrub_stamp":"2024-09-17T05:36:47.067218+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.067218+0000","objects_scrubbed":6,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:07.060712+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":8218,"num_write":88,"num_write_kb":8239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"31'5226","reported_seq":5281,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:56.389076+0000","last_change":"2024-09-17T05:36:56.389008+0000","last_active":"2024-09-17T05:36:56.389076+0000","last_peered":"2024-09-17T05:36:56.389076+0000","last_clean":"2024-09-17T05:36:56.389076+0000","last_became_active":"2024-09-17T05:27:30.678280+0000","last_became_peered":"2024-09-17T05:27:30.678280+0000","last_unstale":"2024-09-17T05:36:56.389076+0000","last_undegraded":"2024-09-17T05:36:56.389076+0000","last_fullsized":"2024-09-17T05:36:56.389076+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5226","last_scrub_stamp":"2024-09-17T05:36:56.388901+0000","last_deep_scrub":"31'5226","last_deep_scrub_stamp":"2024-09-17T05:36:56.388901+0000","last_clean_scrub_stamp":"2024-09-17T05:36:56.388901+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:47:16.036109+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":0,"num_write":76989,"num_write_kb":960359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5,6],"acting":[3,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.661309+0000","last_change":"2024-09-17T05:36:41.661192+0000","last_active":"2024-09-17T05:36:41.661309+0000","last_peered":"2024-09-17T05:36:41.661309+0000","last_clean":"2024-09-17T05:36:41.661309+0000","last_became_active":"2024-09-17T05:27:26.656365+0000","last_became_peered":"2024-09-17T05:27:26.656365+0000","last_unstale":"2024-09-17T05:36:41.661309+0000","last_undegraded":"2024-09-17T05:36:41.661309+0000","last_fullsized":"2024-09-17T05:36:41.661309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:41.661044+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:41.661044+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.661044+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:40:48.551650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"31'221","reported_seq":275,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.103803+0000","last_change":"2024-09-17T05:36:36.402251+0000","last_active":"2024-09-17T05:36:38.103803+0000","last_peered":"2024-09-17T05:36:38.103803+0000","last_clean":"2024-09-17T05:36:38.103803+0000","last_became_active":"2024-09-17T05:27:25.662010+0000","last_became_peered":"2024-09-17T05:27:25.662010+0000","last_unstale":"2024-09-17T05:36:38.103803+0000","last_undegraded":"2024-09-17T05:36:38.103803+0000","last_fullsized":"2024-09-17T05:36:38.103803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'218","last_scrub_stamp":"2024-09-17T05:36:36.402105+0000","last_deep_scrub":"31'218","last_deep_scrub_stamp":"2024-09-17T05:36:36.402105+0000","last_clean_scrub_stamp":"2024-09-17T05:36:36.402105+0000","objects_scrubbed":5,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:36.078027+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":4110,"num_write":213,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"31'5421","reported_seq":5479,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.718298+0000","last_change":"2024-09-17T05:36:46.718223+0000","last_active":"2024-09-17T05:36:46.718298+0000","last_peered":"2024-09-17T05:36:46.718298+0000","last_clean":"2024-09-17T05:36:46.718298+0000","last_became_active":"2024-09-17T05:27:30.679108+0000","last_became_peered":"2024-09-17T05:27:30.679108+0000","last_unstale":"2024-09-17T05:36:46.718298+0000","last_undegraded":"2024-09-17T05:36:46.718298+0000","last_fullsized":"2024-09-17T05:36:46.718298+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5421","last_scrub_stamp":"2024-09-17T05:36:46.718099+0000","last_deep_scrub":"31'5421","last_deep_scrub_stamp":"2024-09-17T05:36:46.718099+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.718099+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:51.089726+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":79593,"num_write_kb":1008039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5,0],"acting":[4,3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.442060+0000","last_change":"2024-09-17T05:36:40.441990+0000","last_active":"2024-09-17T05:36:40.442060+0000","last_peered":"2024-09-17T05:36:40.442060+0000","last_clean":"2024-09-17T05:36:40.442060+0000","last_became_active":"2024-09-17T05:27:26.653583+0000","last_became_peered":"2024-09-17T05:27:26.653583+0000","last_unstale":"2024-09-17T05:36:40.442060+0000","last_undegraded":"2024-09-17T05:36:40.442060+0000","last_fullsized":"2024-09-17T05:36:40.442060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:40.441903+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:40.441903+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.441903+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:24:32.852286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"31'15","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.004345+0000","last_change":"2024-09-17T05:36:43.004273+0000","last_active":"2024-09-17T05:36:43.004345+0000","last_peered":"2024-09-17T05:36:43.004345+0000","last_clean":"2024-09-17T05:36:43.004345+0000","last_became_active":"2024-09-17T05:27:25.658164+0000","last_became_peered":"2024-09-17T05:27:25.658164+0000","last_unstale":"2024-09-17T05:36:43.004345+0000","last_undegraded":"2024-09-17T05:36:43.004345+0000","last_fullsized":"2024-09-17T05:36:43.004345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:43.004159+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:43.004159+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.004159+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:36.614676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"31'5687","reported_seq":5741,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.370300+0000","last_change":"2024-09-17T05:36:49.370233+0000","last_active":"2024-09-17T05:36:49.370300+0000","last_peered":"2024-09-17T05:36:49.370300+0000","last_clean":"2024-09-17T05:36:49.370300+0000","last_became_active":"2024-09-17T05:27:30.677274+0000","last_became_peered":"2024-09-17T05:27:30.677274+0000","last_unstale":"2024-09-17T05:36:49.370300+0000","last_undegraded":"2024-09-17T05:36:49.370300+0000","last_fullsized":"2024-09-17T05:36:49.370300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5687","last_scrub_stamp":"2024-09-17T05:36:49.370130+0000","last_deep_scrub":"31'5687","last_deep_scrub_stamp":"2024-09-17T05:36:49.370130+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.370130+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:01.118430+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":0,"num_write":83481,"num_write_kb":1022097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1,7],"acting":[2,3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"31'16","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.193573+0000","last_change":"2024-09-17T05:36:46.193442+0000","last_active":"2024-09-17T05:36:46.193573+0000","last_peered":"2024-09-17T05:36:46.193573+0000","last_clean":"2024-09-17T05:36:46.193573+0000","last_became_active":"2024-09-17T05:27:26.661505+0000","last_became_peered":"2024-09-17T05:27:26.661505+0000","last_unstale":"2024-09-17T05:36:46.193573+0000","last_undegraded":"2024-09-17T05:36:46.193573+0000","last_fullsized":"2024-09-17T05:36:46.193573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-09-17T05:36:46.193342+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-09-17T05:36:46.193342+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.193342+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:55.328039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"31'6","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.662830+0000","last_change":"2024-09-17T05:36:45.662759+0000","last_active":"2024-09-17T05:36:45.662830+0000","last_peered":"2024-09-17T05:36:45.662830+0000","last_clean":"2024-09-17T05:36:45.662830+0000","last_became_active":"2024-09-17T05:27:25.657992+0000","last_became_peered":"2024-09-17T05:27:25.657992+0000","last_unstale":"2024-09-17T05:36:45.662830+0000","last_undegraded":"2024-09-17T05:36:45.662830+0000","last_fullsized":"2024-09-17T05:36:45.662830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-09-17T05:36:45.662651+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-09-17T05:36:45.662651+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.662651+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:13.241217+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"31'5405","reported_seq":5457,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.081958+0000","last_change":"2024-09-17T05:36:46.081958+0000","last_active":"2024-09-17T05:36:46.081958+0000","last_peered":"2024-09-17T05:36:46.081958+0000","last_clean":"2024-09-17T05:36:46.081958+0000","last_became_active":"2024-09-17T05:27:30.688922+0000","last_became_peered":"2024-09-17T05:27:30.688922+0000","last_unstale":"2024-09-17T05:36:46.081958+0000","last_undegraded":"2024-09-17T05:36:46.081958+0000","last_fullsized":"2024-09-17T05:36:46.081958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5405","last_scrub_stamp":"2024-09-17T05:36:46.081821+0000","last_deep_scrub":"31'5405","last_deep_scrub_stamp":"2024-09-17T05:36:46.081821+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.081821+0000","objects_scrubbed":0,"log_size":5405,"log_dups_size":0,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:48.866580+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":0,"num_write":79725,"num_write_kb":1002963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3,2],"acting":[7,1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.356569+0000","last_change":"2024-09-17T05:36:39.356509+0000","last_active":"2024-09-17T05:36:39.356569+0000","last_peered":"2024-09-17T05:36:39.356569+0000","last_clean":"2024-09-17T05:36:39.356569+0000","last_became_active":"2024-09-17T05:27:26.652669+0000","last_became_peered":"2024-09-17T05:27:26.652669+0000","last_unstale":"2024-09-17T05:36:39.356569+0000","last_undegraded":"2024-09-17T05:36:39.356569+0000","last_fullsized":"2024-09-17T05:36:39.356569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:39.356424+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:39.356424+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.356424+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:44.278824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"31'14","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.739237+0000","last_change":"2024-09-17T05:36:41.739166+0000","last_active":"2024-09-17T05:36:41.739237+0000","last_peered":"2024-09-17T05:36:41.739237+0000","last_clean":"2024-09-17T05:36:41.739237+0000","last_became_active":"2024-09-17T05:27:25.650355+0000","last_became_peered":"2024-09-17T05:27:25.650355+0000","last_unstale":"2024-09-17T05:36:41.739237+0000","last_undegraded":"2024-09-17T05:36:41.739237+0000","last_fullsized":"2024-09-17T05:36:41.739237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:41.739054+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:41.739054+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.739054+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:40.946191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'5609","reported_seq":5665,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:54.671761+0000","last_change":"2024-09-17T05:36:54.671761+0000","last_active":"2024-09-17T05:36:54.671761+0000","last_peered":"2024-09-17T05:36:54.671761+0000","last_clean":"2024-09-17T05:36:54.671761+0000","last_became_active":"2024-09-17T05:27:30.678928+0000","last_became_peered":"2024-09-17T05:27:30.678928+0000","last_unstale":"2024-09-17T05:36:54.671761+0000","last_undegraded":"2024-09-17T05:36:54.671761+0000","last_fullsized":"2024-09-17T05:36:54.671761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5609","last_scrub_stamp":"2024-09-17T05:36:54.671595+0000","last_deep_scrub":"31'5609","last_deep_scrub_stamp":"2024-09-17T05:36:54.671595+0000","last_clean_scrub_stamp":"2024-09-17T05:36:54.671595+0000","objects_scrubbed":0,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:15.270945+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":82713,"num_write_kb":1020969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3,7],"acting":[4,0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6490","reported_seq":6546,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.670705+0000","last_change":"2024-09-17T05:36:43.670705+0000","last_active":"2024-09-17T05:36:43.670705+0000","last_peered":"2024-09-17T05:36:43.670705+0000","last_clean":"2024-09-17T05:36:43.670705+0000","last_became_active":"2024-09-17T05:27:30.677104+0000","last_became_peered":"2024-09-17T05:27:30.677104+0000","last_unstale":"2024-09-17T05:36:43.670705+0000","last_undegraded":"2024-09-17T05:36:43.670705+0000","last_fullsized":"2024-09-17T05:36:43.670705+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6490","last_scrub_stamp":"2024-09-17T05:36:43.670588+0000","last_deep_scrub":"31'6490","last_deep_scrub_stamp":"2024-09-17T05:36:43.670588+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.670588+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:52.614622+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":0,"num_write":95953,"num_write_kb":1202164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5,3],"acting":[4,1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.618934+0000","last_change":"2024-09-17T05:36:53.618805+0000","last_active":"2024-09-17T05:36:53.618934+0000","last_peered":"2024-09-17T05:36:53.618934+0000","last_clean":"2024-09-17T05:36:53.618934+0000","last_became_active":"2024-09-17T05:27:26.653552+0000","last_became_peered":"2024-09-17T05:27:26.653552+0000","last_unstale":"2024-09-17T05:36:53.618934+0000","last_undegraded":"2024-09-17T05:36:53.618934+0000","last_fullsized":"2024-09-17T05:36:53.618934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:53.618649+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:53.618649+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.618649+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:18.914369+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"31'31","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.521807+0000","last_change":"2024-09-17T05:36:47.521707+0000","last_active":"2024-09-17T05:36:47.521807+0000","last_peered":"2024-09-17T05:36:47.521807+0000","last_clean":"2024-09-17T05:36:47.521807+0000","last_became_active":"2024-09-17T05:27:25.649082+0000","last_became_peered":"2024-09-17T05:27:25.649082+0000","last_unstale":"2024-09-17T05:36:47.521807+0000","last_undegraded":"2024-09-17T05:36:47.521807+0000","last_fullsized":"2024-09-17T05:36:47.521807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'31","last_scrub_stamp":"2024-09-17T05:36:47.521587+0000","last_deep_scrub":"31'31","last_deep_scrub_stamp":"2024-09-17T05:36:47.521587+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.521587+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:56.095889+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":25,"num_write_kb":8217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"31'5638","reported_seq":5698,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.463714+0000","last_change":"2024-09-17T05:36:39.463714+0000","last_active":"2024-09-17T05:36:39.463714+0000","last_peered":"2024-09-17T05:36:39.463714+0000","last_clean":"2024-09-17T05:36:39.463714+0000","last_became_active":"2024-09-17T05:27:30.683452+0000","last_became_peered":"2024-09-17T05:27:30.683452+0000","last_unstale":"2024-09-17T05:36:39.463714+0000","last_undegraded":"2024-09-17T05:36:39.463714+0000","last_fullsized":"2024-09-17T05:36:39.463714+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5638","last_scrub_stamp":"2024-09-17T05:36:39.463599+0000","last_deep_scrub":"31'5638","last_deep_scrub_stamp":"2024-09-17T05:36:39.463599+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.463599+0000","objects_scrubbed":0,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:17.221508+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":83062,"num_write_kb":1044536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5,1],"acting":[3,2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"31'19","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.224347+0000","last_change":"2024-09-17T05:36:45.224277+0000","last_active":"2024-09-17T05:36:45.224347+0000","last_peered":"2024-09-17T05:36:45.224347+0000","last_clean":"2024-09-17T05:36:45.224347+0000","last_became_active":"2024-09-17T05:27:26.652888+0000","last_became_peered":"2024-09-17T05:27:26.652888+0000","last_unstale":"2024-09-17T05:36:45.224347+0000","last_undegraded":"2024-09-17T05:36:45.224347+0000","last_fullsized":"2024-09-17T05:36:45.224347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:45.224183+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:45.224183+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.224183+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:59.844340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"31'7","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.039168+0000","last_change":"2024-09-17T05:36:39.039099+0000","last_active":"2024-09-17T05:36:39.039168+0000","last_peered":"2024-09-17T05:36:39.039168+0000","last_clean":"2024-09-17T05:36:39.039168+0000","last_became_active":"2024-09-17T05:27:25.659828+0000","last_became_peered":"2024-09-17T05:27:25.659828+0000","last_unstale":"2024-09-17T05:36:39.039168+0000","last_undegraded":"2024-09-17T05:36:39.039168+0000","last_fullsized":"2024-09-17T05:36:39.039168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:39.039005+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:39.039005+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.039005+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:12:41.557964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"31'25","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:37:00.403824+0000","last_change":"2024-09-17T05:37:00.403824+0000","last_active":"2024-09-17T05:37:00.403824+0000","last_peered":"2024-09-17T05:37:00.403824+0000","last_clean":"2024-09-17T05:37:00.403824+0000","last_became_active":"2024-09-17T05:27:25.654655+0000","last_became_peered":"2024-09-17T05:27:25.654655+0000","last_unstale":"2024-09-17T05:37:00.403824+0000","last_undegraded":"2024-09-17T05:37:00.403824+0000","last_fullsized":"2024-09-17T05:37:00.403824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'25","last_scrub_stamp":"2024-09-17T05:37:00.403738+0000","last_deep_scrub":"31'25","last_deep_scrub_stamp":"2024-09-17T05:37:00.403738+0000","last_clean_scrub_stamp":"2024-09-17T05:37:00.403738+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:12:55.387437+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":4108,"num_write":18,"num_write_kb":4110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.643978+0000","last_change":"2024-09-17T05:36:47.643978+0000","last_active":"2024-09-17T05:36:47.643978+0000","last_peered":"2024-09-17T05:36:47.643978+0000","last_clean":"2024-09-17T05:36:47.643978+0000","last_became_active":"2024-09-17T05:27:26.662932+0000","last_became_peered":"2024-09-17T05:27:26.662932+0000","last_unstale":"2024-09-17T05:36:47.643978+0000","last_undegraded":"2024-09-17T05:36:47.643978+0000","last_fullsized":"2024-09-17T05:36:47.643978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:47.643882+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:47.643882+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.643882+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:33.604932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4609","reported_seq":4655,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.622729+0000","last_change":"2024-09-17T05:36:39.622729+0000","last_active":"2024-09-17T05:36:39.622729+0000","last_peered":"2024-09-17T05:36:39.622729+0000","last_clean":"2024-09-17T05:36:39.622729+0000","last_became_active":"2024-09-17T05:27:30.680416+0000","last_became_peered":"2024-09-17T05:27:30.680416+0000","last_unstale":"2024-09-17T05:36:39.622729+0000","last_undegraded":"2024-09-17T05:36:39.622729+0000","last_fullsized":"2024-09-17T05:36:39.622729+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4609","last_scrub_stamp":"2024-09-17T05:36:39.622630+0000","last_deep_scrub":"31'4609","last_deep_scrub_stamp":"2024-09-17T05:36:39.622630+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.622630+0000","objects_scrubbed":0,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:54.799297+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":68113,"num_write_kb":833128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,7],"acting":[4,6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"31'1238","reported_seq":1287,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.212158+0000","last_change":"2024-09-17T05:36:39.212158+0000","last_active":"2024-09-17T05:36:39.212158+0000","last_peered":"2024-09-17T05:36:39.212158+0000","last_clean":"2024-09-17T05:36:39.212158+0000","last_became_active":"2024-09-17T05:27:25.654353+0000","last_became_peered":"2024-09-17T05:27:25.654353+0000","last_unstale":"2024-09-17T05:36:39.212158+0000","last_undegraded":"2024-09-17T05:36:39.212158+0000","last_fullsized":"2024-09-17T05:36:39.212158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'1238","last_scrub_stamp":"2024-09-17T05:36:39.212032+0000","last_deep_scrub":"31'1238","last_deep_scrub_stamp":"2024-09-17T05:36:39.212032+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.212032+0000","objects_scrubbed":3,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:13.037156+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4196027,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4098,"num_write":1229,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"31'13","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.706309+0000","last_change":"2024-09-17T05:36:38.706309+0000","last_active":"2024-09-17T05:36:38.706309+0000","last_peered":"2024-09-17T05:36:38.706309+0000","last_clean":"2024-09-17T05:36:38.706309+0000","last_became_active":"2024-09-17T05:27:26.657585+0000","last_became_peered":"2024-09-17T05:27:26.657585+0000","last_unstale":"2024-09-17T05:36:38.706309+0000","last_undegraded":"2024-09-17T05:36:38.706309+0000","last_fullsized":"2024-09-17T05:36:38.706309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:38.706208+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:38.706208+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.706208+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:59.688655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"31'6242","reported_seq":6299,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.027356+0000","last_change":"2024-09-17T05:36:49.027356+0000","last_active":"2024-09-17T05:36:49.027356+0000","last_peered":"2024-09-17T05:36:49.027356+0000","last_clean":"2024-09-17T05:36:49.027356+0000","last_became_active":"2024-09-17T05:27:30.684068+0000","last_became_peered":"2024-09-17T05:27:30.684068+0000","last_unstale":"2024-09-17T05:36:49.027356+0000","last_undegraded":"2024-09-17T05:36:49.027356+0000","last_fullsized":"2024-09-17T05:36:49.027356+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6242","last_scrub_stamp":"2024-09-17T05:36:49.027241+0000","last_deep_scrub":"31'6242","last_deep_scrub_stamp":"2024-09-17T05:36:49.027241+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.027241+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:59:05.514832+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":92301,"num_write_kb":1140422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1,0],"acting":[7,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.338972+0000","last_change":"2024-09-17T05:36:47.338972+0000","last_active":"2024-09-17T05:36:47.338972+0000","last_peered":"2024-09-17T05:36:47.338972+0000","last_clean":"2024-09-17T05:36:47.338972+0000","last_became_active":"2024-09-17T05:27:26.649763+0000","last_became_peered":"2024-09-17T05:27:26.649763+0000","last_unstale":"2024-09-17T05:36:47.338972+0000","last_undegraded":"2024-09-17T05:36:47.338972+0000","last_fullsized":"2024-09-17T05:36:47.338972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:47.338887+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:47.338887+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.338887+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:40.881120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"31'13","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.015447+0000","last_change":"2024-09-17T05:36:42.015447+0000","last_active":"2024-09-17T05:36:42.015447+0000","last_peered":"2024-09-17T05:36:42.015447+0000","last_clean":"2024-09-17T05:36:42.015447+0000","last_became_active":"2024-09-17T05:27:26.651967+0000","last_became_peered":"2024-09-17T05:27:26.651967+0000","last_unstale":"2024-09-17T05:36:42.015447+0000","last_undegraded":"2024-09-17T05:36:42.015447+0000","last_fullsized":"2024-09-17T05:36:42.015447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:42.015349+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:42.015349+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.015349+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:30.039576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.22","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:37:02.476689+0000","last_change":"2024-09-17T05:37:02.476689+0000","last_active":"2024-09-17T05:37:02.476689+0000","last_peered":"2024-09-17T05:37:02.476689+0000","last_clean":"2024-09-17T05:37:02.476689+0000","last_became_active":"2024-09-17T05:27:26.660666+0000","last_became_peered":"2024-09-17T05:27:26.660666+0000","last_unstale":"2024-09-17T05:37:02.476689+0000","last_undegraded":"2024-09-17T05:37:02.476689+0000","last_fullsized":"2024-09-17T05:37:02.476689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:37:02.476566+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:37:02.476566+0000","last_clean_scrub_stamp":"2024-09-17T05:37:02.476566+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:05.790740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.177392+0000","last_change":"2024-09-17T05:36:48.177392+0000","last_active":"2024-09-17T05:36:48.177392+0000","last_peered":"2024-09-17T05:36:48.177392+0000","last_clean":"2024-09-17T05:36:48.177392+0000","last_became_active":"2024-09-17T05:27:26.659935+0000","last_became_peered":"2024-09-17T05:27:26.659935+0000","last_unstale":"2024-09-17T05:36:48.177392+0000","last_undegraded":"2024-09-17T05:36:48.177392+0000","last_fullsized":"2024-09-17T05:36:48.177392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'11","last_scrub_stamp":"2024-09-17T05:36:48.177304+0000","last_deep_scrub":"31'11","last_deep_scrub_stamp":"2024-09-17T05:36:48.177304+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.177304+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:34:49.273171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.24","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.174036+0000","last_change":"2024-09-17T05:36:48.174036+0000","last_active":"2024-09-17T05:36:48.174036+0000","last_peered":"2024-09-17T05:36:48.174036+0000","last_clean":"2024-09-17T05:36:48.174036+0000","last_became_active":"2024-09-17T05:27:26.660695+0000","last_became_peered":"2024-09-17T05:27:26.660695+0000","last_unstale":"2024-09-17T05:36:48.174036+0000","last_undegraded":"2024-09-17T05:36:48.174036+0000","last_fullsized":"2024-09-17T05:36:48.174036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:48.173944+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:48.173944+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.173944+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:57.117978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:43.114788+0000","last_change":"2024-09-17T05:36:43.114788+0000","last_active":"2024-09-17T05:36:43.114788+0000","last_peered":"2024-09-17T05:36:43.114788+0000","last_clean":"2024-09-17T05:36:43.114788+0000","last_became_active":"2024-09-17T05:27:26.656014+0000","last_became_peered":"2024-09-17T05:27:26.656014+0000","last_unstale":"2024-09-17T05:36:43.114788+0000","last_undegraded":"2024-09-17T05:36:43.114788+0000","last_fullsized":"2024-09-17T05:36:43.114788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:43.114691+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:43.114691+0000","last_clean_scrub_stamp":"2024-09-17T05:36:43.114691+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:50.140756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.26","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:40.392372+0000","last_change":"2024-09-17T05:36:40.392372+0000","last_active":"2024-09-17T05:36:40.392372+0000","last_peered":"2024-09-17T05:36:40.392372+0000","last_clean":"2024-09-17T05:36:40.392372+0000","last_became_active":"2024-09-17T05:27:26.655949+0000","last_became_peered":"2024-09-17T05:27:26.655949+0000","last_unstale":"2024-09-17T05:36:40.392372+0000","last_undegraded":"2024-09-17T05:36:40.392372+0000","last_fullsized":"2024-09-17T05:36:40.392372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:40.392280+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:40.392280+0000","last_clean_scrub_stamp":"2024-09-17T05:36:40.392280+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:53:44.758222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:50.252032+0000","last_change":"2024-09-17T05:36:50.252032+0000","last_active":"2024-09-17T05:36:50.252032+0000","last_peered":"2024-09-17T05:36:50.252032+0000","last_clean":"2024-09-17T05:36:50.252032+0000","last_became_active":"2024-09-17T05:27:26.658942+0000","last_became_peered":"2024-09-17T05:27:26.658942+0000","last_unstale":"2024-09-17T05:36:50.252032+0000","last_undegraded":"2024-09-17T05:36:50.252032+0000","last_fullsized":"2024-09-17T05:36:50.252032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:50.251929+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:50.251929+0000","last_clean_scrub_stamp":"2024-09-17T05:36:50.251929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:34.342583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"31'12","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.462043+0000","last_change":"2024-09-17T05:36:48.462043+0000","last_active":"2024-09-17T05:36:48.462043+0000","last_peered":"2024-09-17T05:36:48.462043+0000","last_clean":"2024-09-17T05:36:48.462043+0000","last_became_active":"2024-09-17T05:27:26.661199+0000","last_became_peered":"2024-09-17T05:27:26.661199+0000","last_unstale":"2024-09-17T05:36:48.462043+0000","last_undegraded":"2024-09-17T05:36:48.462043+0000","last_fullsized":"2024-09-17T05:36:48.462043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:48.461953+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:48.461953+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.461953+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:12.821365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.204218+0000","last_change":"2024-09-17T05:36:49.204218+0000","last_active":"2024-09-17T05:36:49.204218+0000","last_peered":"2024-09-17T05:36:49.204218+0000","last_clean":"2024-09-17T05:36:49.204218+0000","last_became_active":"2024-09-17T05:27:26.655243+0000","last_became_peered":"2024-09-17T05:27:26.655243+0000","last_unstale":"2024-09-17T05:36:49.204218+0000","last_undegraded":"2024-09-17T05:36:49.204218+0000","last_fullsized":"2024-09-17T05:36:49.204218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:49.204112+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:49.204112+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.204112+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:08.061956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2a","version":"31'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:39.730554+0000","last_change":"2024-09-17T05:36:39.730554+0000","last_active":"2024-09-17T05:36:39.730554+0000","last_peered":"2024-09-17T05:36:39.730554+0000","last_clean":"2024-09-17T05:36:39.730554+0000","last_became_active":"2024-09-17T05:27:26.663377+0000","last_became_peered":"2024-09-17T05:27:26.663377+0000","last_unstale":"2024-09-17T05:36:39.730554+0000","last_undegraded":"2024-09-17T05:36:39.730554+0000","last_fullsized":"2024-09-17T05:36:39.730554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:39.730431+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:39.730431+0000","last_clean_scrub_stamp":"2024-09-17T05:36:39.730431+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:40.179412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"31'17","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.684772+0000","last_change":"2024-09-17T05:36:48.684772+0000","last_active":"2024-09-17T05:36:48.684772+0000","last_peered":"2024-09-17T05:36:48.684772+0000","last_clean":"2024-09-17T05:36:48.684772+0000","last_became_active":"2024-09-17T05:27:26.652684+0000","last_became_peered":"2024-09-17T05:27:26.652684+0000","last_unstale":"2024-09-17T05:36:48.684772+0000","last_undegraded":"2024-09-17T05:36:48.684772+0000","last_fullsized":"2024-09-17T05:36:48.684772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'17","last_scrub_stamp":"2024-09-17T05:36:48.684691+0000","last_deep_scrub":"31'17","last_deep_scrub_stamp":"2024-09-17T05:36:48.684691+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.684691+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:15.592178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"31'11","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.140037+0000","last_change":"2024-09-17T05:36:37.610072+0000","last_active":"2024-09-17T05:36:38.140037+0000","last_peered":"2024-09-17T05:36:38.140037+0000","last_clean":"2024-09-17T05:36:38.140037+0000","last_became_active":"2024-09-17T05:27:26.659142+0000","last_became_peered":"2024-09-17T05:27:26.659142+0000","last_unstale":"2024-09-17T05:36:38.140037+0000","last_undegraded":"2024-09-17T05:36:38.140037+0000","last_fullsized":"2024-09-17T05:36:38.140037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:37.609959+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:37.609959+0000","last_clean_scrub_stamp":"2024-09-17T05:36:37.609959+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:03.953462+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"31'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:54.412079+0000","last_change":"2024-09-17T05:36:54.412079+0000","last_active":"2024-09-17T05:36:54.412079+0000","last_peered":"2024-09-17T05:36:54.412079+0000","last_clean":"2024-09-17T05:36:54.412079+0000","last_became_active":"2024-09-17T05:27:26.656627+0000","last_became_peered":"2024-09-17T05:27:26.656627+0000","last_unstale":"2024-09-17T05:36:54.412079+0000","last_undegraded":"2024-09-17T05:36:54.412079+0000","last_fullsized":"2024-09-17T05:36:54.412079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:54.411925+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:54.411925+0000","last_clean_scrub_stamp":"2024-09-17T05:36:54.411925+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:44.212757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"31'19","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:45.004268+0000","last_change":"2024-09-17T05:36:45.004268+0000","last_active":"2024-09-17T05:36:45.004268+0000","last_peered":"2024-09-17T05:36:45.004268+0000","last_clean":"2024-09-17T05:36:45.004268+0000","last_became_active":"2024-09-17T05:27:26.656577+0000","last_became_peered":"2024-09-17T05:27:26.656577+0000","last_unstale":"2024-09-17T05:36:45.004268+0000","last_undegraded":"2024-09-17T05:36:45.004268+0000","last_fullsized":"2024-09-17T05:36:45.004268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'19","last_scrub_stamp":"2024-09-17T05:36:45.004163+0000","last_deep_scrub":"31'19","last_deep_scrub_stamp":"2024-09-17T05:36:45.004163+0000","last_clean_scrub_stamp":"2024-09-17T05:36:45.004163+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:00:37.747058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2f","version":"31'13","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.390514+0000","last_change":"2024-09-17T05:36:38.390411+0000","last_active":"2024-09-17T05:36:38.390514+0000","last_peered":"2024-09-17T05:36:38.390514+0000","last_clean":"2024-09-17T05:36:38.390514+0000","last_became_active":"2024-09-17T05:27:26.661632+0000","last_became_peered":"2024-09-17T05:27:26.661632+0000","last_unstale":"2024-09-17T05:36:38.390514+0000","last_undegraded":"2024-09-17T05:36:38.390514+0000","last_fullsized":"2024-09-17T05:36:38.390514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:38.390300+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:38.390300+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.390300+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:21.127766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"31'12","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:49.462921+0000","last_change":"2024-09-17T05:36:49.462851+0000","last_active":"2024-09-17T05:36:49.462921+0000","last_peered":"2024-09-17T05:36:49.462921+0000","last_clean":"2024-09-17T05:36:49.462921+0000","last_became_active":"2024-09-17T05:27:26.659112+0000","last_became_peered":"2024-09-17T05:27:26.659112+0000","last_unstale":"2024-09-17T05:36:49.462921+0000","last_undegraded":"2024-09-17T05:36:49.462921+0000","last_fullsized":"2024-09-17T05:36:49.462921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:49.462764+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:49.462764+0000","last_clean_scrub_stamp":"2024-09-17T05:36:49.462764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:53.324712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.31","version":"31'14","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:51.290552+0000","last_change":"2024-09-17T05:36:51.290486+0000","last_active":"2024-09-17T05:36:51.290552+0000","last_peered":"2024-09-17T05:36:51.290552+0000","last_clean":"2024-09-17T05:36:51.290552+0000","last_became_active":"2024-09-17T05:27:26.662933+0000","last_became_peered":"2024-09-17T05:27:26.662933+0000","last_unstale":"2024-09-17T05:36:51.290552+0000","last_undegraded":"2024-09-17T05:36:51.290552+0000","last_fullsized":"2024-09-17T05:36:51.290552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-09-17T05:36:51.290399+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-09-17T05:36:51.290399+0000","last_clean_scrub_stamp":"2024-09-17T05:36:51.290399+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:44:51.991292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"31'17","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.119537+0000","last_change":"2024-09-17T05:36:46.119431+0000","last_active":"2024-09-17T05:36:46.119537+0000","last_peered":"2024-09-17T05:36:46.119537+0000","last_clean":"2024-09-17T05:36:46.119537+0000","last_became_active":"2024-09-17T05:27:26.656381+0000","last_became_peered":"2024-09-17T05:27:26.656381+0000","last_unstale":"2024-09-17T05:36:46.119537+0000","last_undegraded":"2024-09-17T05:36:46.119537+0000","last_fullsized":"2024-09-17T05:36:46.119537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'17","last_scrub_stamp":"2024-09-17T05:36:46.119291+0000","last_deep_scrub":"31'17","last_deep_scrub_stamp":"2024-09-17T05:36:46.119291+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.119291+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:15.011798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"31'15","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:47.147176+0000","last_change":"2024-09-17T05:36:47.147113+0000","last_active":"2024-09-17T05:36:47.147176+0000","last_peered":"2024-09-17T05:36:47.147176+0000","last_clean":"2024-09-17T05:36:47.147176+0000","last_became_active":"2024-09-17T05:27:26.659936+0000","last_became_peered":"2024-09-17T05:27:26.659936+0000","last_unstale":"2024-09-17T05:36:47.147176+0000","last_undegraded":"2024-09-17T05:36:47.147176+0000","last_fullsized":"2024-09-17T05:36:47.147176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:47.147028+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:47.147028+0000","last_clean_scrub_stamp":"2024-09-17T05:36:47.147028+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:19.880014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:38.610869+0000","last_change":"2024-09-17T05:36:38.610809+0000","last_active":"2024-09-17T05:36:38.610869+0000","last_peered":"2024-09-17T05:36:38.610869+0000","last_clean":"2024-09-17T05:36:38.610869+0000","last_became_active":"2024-09-17T05:27:26.659487+0000","last_became_peered":"2024-09-17T05:27:26.659487+0000","last_unstale":"2024-09-17T05:36:38.610869+0000","last_undegraded":"2024-09-17T05:36:38.610869+0000","last_fullsized":"2024-09-17T05:36:38.610869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:38.610732+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:38.610732+0000","last_clean_scrub_stamp":"2024-09-17T05:36:38.610732+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:32.152965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.675524+0000","last_change":"2024-09-17T05:36:42.675441+0000","last_active":"2024-09-17T05:36:42.675524+0000","last_peered":"2024-09-17T05:36:42.675524+0000","last_clean":"2024-09-17T05:36:42.675524+0000","last_became_active":"2024-09-17T05:27:26.665245+0000","last_became_peered":"2024-09-17T05:27:26.665245+0000","last_unstale":"2024-09-17T05:36:42.675524+0000","last_undegraded":"2024-09-17T05:36:42.675524+0000","last_fullsized":"2024-09-17T05:36:42.675524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:42.675339+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:42.675339+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.675339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:09.582826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"31'13","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:46.696572+0000","last_change":"2024-09-17T05:36:46.696505+0000","last_active":"2024-09-17T05:36:46.696572+0000","last_peered":"2024-09-17T05:36:46.696572+0000","last_clean":"2024-09-17T05:36:46.696572+0000","last_became_active":"2024-09-17T05:27:26.657852+0000","last_became_peered":"2024-09-17T05:27:26.657852+0000","last_unstale":"2024-09-17T05:36:46.696572+0000","last_undegraded":"2024-09-17T05:36:46.696572+0000","last_fullsized":"2024-09-17T05:36:46.696572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-09-17T05:36:46.696388+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-09-17T05:36:46.696388+0000","last_clean_scrub_stamp":"2024-09-17T05:36:46.696388+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:09.445934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"31'16","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.760881+0000","last_change":"2024-09-17T05:36:42.760807+0000","last_active":"2024-09-17T05:36:42.760881+0000","last_peered":"2024-09-17T05:36:42.760881+0000","last_clean":"2024-09-17T05:36:42.760881+0000","last_became_active":"2024-09-17T05:27:26.662046+0000","last_became_peered":"2024-09-17T05:27:26.662046+0000","last_unstale":"2024-09-17T05:36:42.760881+0000","last_undegraded":"2024-09-17T05:36:42.760881+0000","last_fullsized":"2024-09-17T05:36:42.760881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-09-17T05:36:42.760720+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-09-17T05:36:42.760720+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.760720+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:30.784948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"31'5","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:52.310258+0000","last_change":"2024-09-17T05:36:52.310258+0000","last_active":"2024-09-17T05:36:52.310258+0000","last_peered":"2024-09-17T05:36:52.310258+0000","last_clean":"2024-09-17T05:36:52.310258+0000","last_became_active":"2024-09-17T05:27:26.662751+0000","last_became_peered":"2024-09-17T05:27:26.662751+0000","last_unstale":"2024-09-17T05:36:52.310258+0000","last_undegraded":"2024-09-17T05:36:52.310258+0000","last_fullsized":"2024-09-17T05:36:52.310258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-09-17T05:36:52.310114+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-09-17T05:36:52.310114+0000","last_clean_scrub_stamp":"2024-09-17T05:36:52.310114+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:39.798350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"31'9","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.661839+0000","last_change":"2024-09-17T05:36:44.661839+0000","last_active":"2024-09-17T05:36:44.661839+0000","last_peered":"2024-09-17T05:36:44.661839+0000","last_clean":"2024-09-17T05:36:44.661839+0000","last_became_active":"2024-09-17T05:27:26.658056+0000","last_became_peered":"2024-09-17T05:27:26.658056+0000","last_unstale":"2024-09-17T05:36:44.661839+0000","last_undegraded":"2024-09-17T05:36:44.661839+0000","last_fullsized":"2024-09-17T05:36:44.661839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-09-17T05:36:44.661741+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-09-17T05:36:44.661741+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.661741+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:05.214263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"31'10","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:42.384149+0000","last_change":"2024-09-17T05:36:42.384149+0000","last_active":"2024-09-17T05:36:42.384149+0000","last_peered":"2024-09-17T05:36:42.384149+0000","last_clean":"2024-09-17T05:36:42.384149+0000","last_became_active":"2024-09-17T05:27:26.654697+0000","last_became_peered":"2024-09-17T05:27:26.654697+0000","last_unstale":"2024-09-17T05:36:42.384149+0000","last_undegraded":"2024-09-17T05:36:42.384149+0000","last_fullsized":"2024-09-17T05:36:42.384149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-09-17T05:36:42.384012+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-09-17T05:36:42.384012+0000","last_clean_scrub_stamp":"2024-09-17T05:36:42.384012+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:57.560680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"31'15","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:48.645068+0000","last_change":"2024-09-17T05:36:48.645068+0000","last_active":"2024-09-17T05:36:48.645068+0000","last_peered":"2024-09-17T05:36:48.645068+0000","last_clean":"2024-09-17T05:36:48.645068+0000","last_became_active":"2024-09-17T05:27:26.662797+0000","last_became_peered":"2024-09-17T05:27:26.662797+0000","last_unstale":"2024-09-17T05:36:48.645068+0000","last_undegraded":"2024-09-17T05:36:48.645068+0000","last_fullsized":"2024-09-17T05:36:48.645068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'15","last_scrub_stamp":"2024-09-17T05:36:48.644964+0000","last_deep_scrub":"31'15","last_deep_scrub_stamp":"2024-09-17T05:36:48.644964+0000","last_clean_scrub_stamp":"2024-09-17T05:36:48.644964+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:09.972427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"31'7","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:41.348905+0000","last_change":"2024-09-17T05:36:41.348905+0000","last_active":"2024-09-17T05:36:41.348905+0000","last_peered":"2024-09-17T05:36:41.348905+0000","last_clean":"2024-09-17T05:36:41.348905+0000","last_became_active":"2024-09-17T05:27:26.655451+0000","last_became_peered":"2024-09-17T05:27:26.655451+0000","last_unstale":"2024-09-17T05:36:41.348905+0000","last_undegraded":"2024-09-17T05:36:41.348905+0000","last_fullsized":"2024-09-17T05:36:41.348905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'7","last_scrub_stamp":"2024-09-17T05:36:41.348751+0000","last_deep_scrub":"31'7","last_deep_scrub_stamp":"2024-09-17T05:36:41.348751+0000","last_clean_scrub_stamp":"2024-09-17T05:36:41.348751+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:08.318810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"31'12","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:53.316305+0000","last_change":"2024-09-17T05:36:53.316305+0000","last_active":"2024-09-17T05:36:53.316305+0000","last_peered":"2024-09-17T05:36:53.316305+0000","last_clean":"2024-09-17T05:36:53.316305+0000","last_became_active":"2024-09-17T05:27:26.657314+0000","last_became_peered":"2024-09-17T05:27:26.657314+0000","last_unstale":"2024-09-17T05:36:53.316305+0000","last_undegraded":"2024-09-17T05:36:53.316305+0000","last_fullsized":"2024-09-17T05:36:53.316305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-09-17T05:36:53.316209+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-09-17T05:36:53.316209+0000","last_clean_scrub_stamp":"2024-09-17T05:36:53.316209+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:42:23.795879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3e","version":"31'6","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:54.353964+0000","last_change":"2024-09-17T05:36:54.353964+0000","last_active":"2024-09-17T05:36:54.353964+0000","last_peered":"2024-09-17T05:36:54.353964+0000","last_clean":"2024-09-17T05:36:54.353964+0000","last_became_active":"2024-09-17T05:27:26.654690+0000","last_became_peered":"2024-09-17T05:27:26.654690+0000","last_unstale":"2024-09-17T05:36:54.353964+0000","last_undegraded":"2024-09-17T05:36:54.353964+0000","last_fullsized":"2024-09-17T05:36:54.353964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-09-17T05:36:54.353868+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-09-17T05:36:54.353868+0000","last_clean_scrub_stamp":"2024-09-17T05:36:54.353868+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:03:03.509468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3f","version":"31'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T05:36:44.004419+0000","last_change":"2024-09-17T05:36:44.004419+0000","last_active":"2024-09-17T05:36:44.004419+0000","last_peered":"2024-09-17T05:36:44.004419+0000","last_clean":"2024-09-17T05:36:44.004419+0000","last_became_active":"2024-09-17T05:28:07.483003+0000","last_became_peered":"2024-09-17T05:28:07.483003+0000","last_unstale":"2024-09-17T05:36:44.004419+0000","last_undegraded":"2024-09-17T05:36:44.004419+0000","last_fullsized":"2024-09-17T05:36:44.004419+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-09-17T05:36:44.004318+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-09-17T05:36:44.004318+0000","last_clean_scrub_stamp":"2024-09-17T05:36:44.004318+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:02.130745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":525,"num_read_kb":0,"num_write":2621864,"num_write_kb":32721427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177787,"ondisk_log_size":177787,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":82273271,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":349,"num_read_kb":78313,"num_write":2736,"num_write_kb":82427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15000,"num_omap_keys":30,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34103296,"data_stored":164546542,"data_compressed":18858484,"data_compressed_allocated":21905408,"data_compressed_original":152502272,"omap_allocated":0,"internal_metadata":0},"log_size":2708,"ondisk_log_size":2708,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64974,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509560,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215208,"kb_used_data":5976,"kb_used_omap":1,"kb_used_meta":1209214,"kb_avail":93156632,"statfs":{"total":96636764160,"available":95392391168,"internally_reserved":0,"allocated":6119424,"data_stored":37895144,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":1593,"internal_metadata":1238235591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509562,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":844032,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":840830,"kb_avail":93527808,"statfs":{"total":96636764160,"available":95772475392,"internally_reserved":0,"allocated":3260416,"data_stored":12727699,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":1595,"internal_metadata":861010373},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509562,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1132012,"kb_used_data":4508,"kb_used_omap":1,"kb_used_meta":1127486,"kb_avail":93239828,"statfs":{"total":96636764160,"available":95477583872,"internally_reserved":0,"allocated":4616192,"data_stored":24186871,"data_compressed":2785195,"data_compressed_allocated":3256320,"data_compressed_original":23027712,"omap_allocated":1595,"internal_metadata":1154546117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509563,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736580,"kb_used_data":5236,"kb_used_omap":1,"kb_used_meta":731326,"kb_avail":93635260,"statfs":{"total":96636764160,"available":95882506240,"internally_reserved":0,"allocated":5361664,"data_stored":27793064,"data_compressed":3270621,"data_compressed_allocated":3801088,"data_compressed_original":26439680,"omap_allocated":1595,"internal_metadata":748878277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542264,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129504,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":1127038,"kb_avail":93242336,"statfs":{"total":96636764160,"available":95480152064,"internally_reserved":0,"allocated":2506752,"data_stored":12822170,"data_compressed":1499845,"data_compressed_allocated":1761280,"data_compressed_original":12292096,"omap_allocated":1590,"internal_metadata":1154087370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542265,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1086856,"kb_used_data":4664,"kb_used_omap":1,"kb_used_meta":1082174,"kb_avail":93284984,"statfs":{"total":96636764160,"available":95523823616,"internally_reserved":0,"allocated":4775936,"data_stored":12729298,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":1594,"internal_metadata":1108146630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542264,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":916820,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":913726,"kb_avail":93455020,"statfs":{"total":96636764160,"available":95697940480,"internally_reserved":0,"allocated":3149824,"data_stored":12729233,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":1594,"internal_metadata":935655878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574970,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":880292,"kb_used_data":6932,"kb_used_omap":1,"kb_used_meta":873342,"kb_avail":93491548,"statfs":{"total":96636764160,"available":95735345152,"internally_reserved":0,"allocated":7098368,"data_stored":25996447,"data_compressed":2718385,"data_compressed_allocated":3203072,"data_compressed_original":22315008,"omap_allocated":1595,"internal_metadata":894302661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32487,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5029888,"data_stored":27648983,"data_compressed":3270621,"data_compressed_allocated":3801088,"data_compressed_original":26439680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":23452422,"data_compressed":2752708,"data_compressed_allocated":3211264,"data_compressed_original":22454272,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":12583618,"data_compressed":1517711,"data_compressed_allocated":1785856,"data_compressed_original":11452416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5787648,"data_stored":37751063,"data_compressed":4364354,"data_compressed_allocated":4956160,"data_compressed_original":36933632,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":12585152,"data_compressed":1565611,"data_compressed_allocated":1851392,"data_compressed_original":11657216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6701056,"data_stored":25261998,"data_compressed":2685898,"data_compressed_allocated":3158016,"data_compressed_original":21741568,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4444160,"data_stored":12585217,"data_compressed":1201736,"data_compressed_allocated":1380352,"data_compressed_original":9531392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":12678089,"data_compressed":1499845,"data_compressed_allocated":1761280,"data_compressed_original":12292096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:37:20.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 -- 172.21.15.117:0/3784138575 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec0c5660 msgr2=0x7fc4ec071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/3784138575 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec0c5660 0x7fc4ec071c60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4005fd0 tx=0x7fc4e402a820 comp rx=0 tx=0).stop 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 -- 172.21.15.117:0/3784138575 shutdown_connections 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/3784138575 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec0c5660 0x7fc4ec071c60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/3784138575 >> v2:172.21.15.119:3300/0 conn(0x7fc4ec0c67a0 0x7fc4ec152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 -- 172.21.15.117:0/3784138575 >> 172.21.15.117:0/3784138575 conn(0x7fc4ec06e590 msgr2=0x7fc4ec0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 -- 172.21.15.117:0/3784138575 shutdown_connections 2024-09-17T05:37:20.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.608+0000 7fc4f25c6640 1 -- 172.21.15.117:0/3784138575 wait complete. 2024-09-17T05:37:20.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.610+0000 7fc4f25c6640 1 Processor -- start 2024-09-17T05:37:20.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.610+0000 7fc4f25c6640 1 -- start start 2024-09-17T05:37:20.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f25c6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec071850 0x7fc4ec1f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f25c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 0x7fc4ec1f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f25c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec0c67a0 0x7fc4ec147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f25c6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec0c7ff0 con 0x7fc4ec0c5660 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f25c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec0ce0d0 con 0x7fc4ec0c67a0 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f25c6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ec0c83a0 con 0x7fc4ec071850 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 0x7fc4ec1f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 0x7fc4ec1f2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51788/0 (socket says 172.21.15.117:51788) 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 -- 172.21.15.117:0/1941951544 learned_addr learned my addr 172.21.15.117:0/1941951544 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4ebfff640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec071850 0x7fc4ec1f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4f0b63640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec0c67a0 0x7fc4ec147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 -- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec071850 msgr2=0x7fc4ec1f5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec071850 0x7fc4ec1f5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 -- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec0c67a0 msgr2=0x7fc4ec147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec0c67a0 0x7fc4ec147500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 -- 172.21.15.117:0/1941951544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec0cc4a0 con 0x7fc4ec0c5660 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.611+0000 7fc4eb7fe640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 0x7fc4ec1f2890 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00c3d0 tx=0x7fc4dc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.612+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc011790 con 0x7fc4ec0c5660 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.612+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4dc017070 con 0x7fc4ec0c5660 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.612+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec0cc780 con 0x7fc4ec0c5660 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.612+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc011cc0 con 0x7fc4ec0c5660 2024-09-17T05:37:20.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.612+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec1f56a0 con 0x7fc4ec0c5660 2024-09-17T05:37:20.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.613+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ec0cc1f0 con 0x7fc4ec0c5660 2024-09-17T05:37:20.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.614+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc4dc01c460 con 0x7fc4ec0c5660 2024-09-17T05:37:20.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.614+0000 7fc4e8ff9640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4cc078bc0 0x7fc4cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.614+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7fc4dc09b7b0 con 0x7fc4ec0c5660 2024-09-17T05:37:20.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.614+0000 7fc4ebfff640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4cc078bc0 0x7fc4cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.615+0000 7fc4ebfff640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4cc078bc0 0x7fc4cc07b0d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40095f0 tx=0x7fc4d4005f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:20.619 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.618+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc4dc063240 con 0x7fc4ec0c5660 2024-09-17T05:37:20.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.804+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fc4ec0c7940 con 0x7fc4ec0c5660 2024-09-17T05:37:20.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.808+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v37)=0 v37) ==== 123+0+0 (secure 0 0 0) 0x7fc4dc067c70 con 0x7fc4ec0c5660 2024-09-17T05:37:20.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.810+0000 7fc4e8ff9640 1 -- 172.21.15.117:0/1941951544 <== mon.0 v2:172.21.15.117:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fc4dc062d50 con 0x7fc4ec0c5660 2024-09-17T05:37:20.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4cc078bc0 msgr2=0x7fc4cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4cc078bc0 0x7fc4cc07b0d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40095f0 tx=0x7fc4d4005f90 comp rx=0 tx=0).stop 2024-09-17T05:37:20.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 msgr2=0x7fc4ec1f2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 0x7fc4ec1f2890 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00c3d0 tx=0x7fc4dc00c890 comp rx=0 tx=0).stop 2024-09-17T05:37:20.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 shutdown_connections 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fc4cc078bc0 0x7fc4cc07b0d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec0c67a0 0x7fc4ec147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4ec0c5660 0x7fc4ec1f2890 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 --2- 172.21.15.117:0/1941951544 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4ec071850 0x7fc4ec1f5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 >> 172.21.15.117:0/1941951544 conn(0x7fc4ec06e590 msgr2=0x7fc4ec148250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 shutdown_connections 2024-09-17T05:37:20.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.811+0000 7fc4f25c6640 1 -- 172.21.15.117:0/1941951544 wait complete. 2024-09-17T05:37:20.823 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T05:37:20.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:37:20.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d4508b640 1 -- 172.21.15.117:0/113916919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d2c004e60 msgr2=0x7f0d2c0052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d4508b640 1 --2- 172.21.15.117:0/113916919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d2c004e60 0x7f0d2c0052b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0d38001dc0 tx=0x7f0d2c005830 comp rx=0 tx=0).stop 2024-09-17T05:37:20.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d3f59e640 1 -- 172.21.15.117:0/113916919 reap_dead start 2024-09-17T05:37:20.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d4508b640 1 -- 172.21.15.117:0/113916919 shutdown_connections 2024-09-17T05:37:20.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d4508b640 1 -- 172.21.15.117:0/113916919 >> 172.21.15.117:0/113916919 conn(0x7f0d4006e590 msgr2=0x7f0d400cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:20.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d4508b640 1 -- 172.21.15.117:0/113916919 shutdown_connections 2024-09-17T05:37:20.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.938+0000 7f0d4508b640 1 -- 172.21.15.117:0/113916919 wait complete. 2024-09-17T05:37:20.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.940+0000 7f0d4508b640 1 Processor -- start 2024-09-17T05:37:20.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.941+0000 7f0d4508b640 1 -- start start 2024-09-17T05:37:20.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d4508b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d2c004e60 0x7f0d401465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d4508b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 0x7f0d40143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d4508b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d401443b0 0x7f0d40149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d4508b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0d400ce0d0 con 0x7f0d400cb530 2024-09-17T05:37:20.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d4508b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0d400c58a0 con 0x7f0d2c004e60 2024-09-17T05:37:20.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d4508b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f0d400c7ff0 con 0x7f0d401443b0 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 0x7f0d40143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 0x7f0d40143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51814/0 (socket says 172.21.15.117:51814) 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 -- 172.21.15.117:0/1039409151 learned_addr learned my addr 172.21.15.117:0/1039409151 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3f59e640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d401443b0 0x7f0d40149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3ed9d640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d2c004e60 0x7f0d401465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 -- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d401443b0 msgr2=0x7f0d40149450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d401443b0 0x7f0d40149450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 -- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d2c004e60 msgr2=0x7f0d401465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d2c004e60 0x7f0d401465c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 -- 172.21.15.117:0/1039409151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d401515f0 con 0x7f0d400cb530 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3ed9d640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d2c004e60 0x7f0d401465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.942+0000 7f0d3e59c640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 0x7f0d40143e70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0d28004650 tx=0x7f0d2800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:20.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.943+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28010850 con 0x7f0d400cb530 2024-09-17T05:37:20.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.943+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f0d280109f0 con 0x7f0d400cb530 2024-09-17T05:37:20.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.943+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d400ccaf0 con 0x7f0d400cb530 2024-09-17T05:37:20.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.943+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28010e40 con 0x7f0d400cb530 2024-09-17T05:37:20.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.943+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d400ccd90 con 0x7f0d400cb530 2024-09-17T05:37:20.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.944+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d40151890 con 0x7f0d400cb530 2024-09-17T05:37:20.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.944+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d28023440 con 0x7f0d400cb530 2024-09-17T05:37:20.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.945+0000 7f0d377fe640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d20078b90 0x7f0d2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:20.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.945+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f0d2809bfa0 con 0x7f0d400cb530 2024-09-17T05:37:20.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.945+0000 7f0d3ed9d640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d20078b90 0x7f0d2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:20.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.945+0000 7f0d3ed9d640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d20078b90 0x7f0d2007b0a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0d30006d50 tx=0x7f0d300067a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:20.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:20.949+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d28063fb0 con 0x7f0d400cb530 2024-09-17T05:37:21.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.169+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d400c4750 con 0x7f0d400cb530 2024-09-17T05:37:21.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.170+0000 7f0d377fe640 1 -- 172.21.15.117:0/1039409151 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6894 (secure 0 0 0) 0x7f0d280638b0 con 0x7f0d400cb530 2024-09-17T05:37:21.171 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:37:21.171 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":44,"btime":"2024-09-17T05:35:55:610284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5141,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/714171389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":714171389},{"type":"v1","addr":"172.21.15.119:6839","nonce":714171389}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:45.569656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:active","state_seq":20,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5457,"qdb_cluster":[5457]},"id":1}]} 2024-09-17T05:37:21.171 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 44 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d20078b90 msgr2=0x7f0d2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d20078b90 0x7f0d2007b0a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0d30006d50 tx=0x7f0d300067a0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 msgr2=0x7f0d40143e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 0x7f0d40143e70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0d28004650 tx=0x7f0d2800ba40 comp rx=0 tx=0).stop 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 shutdown_connections 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f0d20078b90 0x7f0d2007b0a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0d401443b0 0x7f0d40149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d400cb530 0x7f0d40143e70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 --2- 172.21.15.117:0/1039409151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0d2c004e60 0x7f0d401465c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 >> 172.21.15.117:0/1039409151 conn(0x7f0d4006e590 msgr2=0x7f0d400c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:21.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 shutdown_connections 2024-09-17T05:37:21.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.173+0000 7f0d4508b640 1 -- 172.21.15.117:0/1039409151 wait complete. 2024-09-17T05:37:21.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 -- 172.21.15.117:0/738050968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c152210 msgr2=0x7f8c9c0c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 --2- 172.21.15.117:0/738050968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c152210 0x7f8c9c0c4f50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0c4b20 tx=0x7f8c9802a4a0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 -- 172.21.15.117:0/738050968 shutdown_connections 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 --2- 172.21.15.117:0/738050968 >> v2:172.21.15.117:3300/0 conn(0x7f8c9c0c5490 0x7f8c9c162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 --2- 172.21.15.117:0/738050968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c152210 0x7f8c9c0c4f50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 -- 172.21.15.117:0/738050968 >> 172.21.15.117:0/738050968 conn(0x7f8c9c06e590 msgr2=0x7f8c9c0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:21.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 -- 172.21.15.117:0/738050968 shutdown_connections 2024-09-17T05:37:21.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.311+0000 7f8ca390b640 1 -- 172.21.15.117:0/738050968 wait complete. 2024-09-17T05:37:21.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.313+0000 7f8ca390b640 1 Processor -- start 2024-09-17T05:37:21.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.313+0000 7f8ca390b640 1 -- start start 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca390b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c0c5490 0x7f8c9c14a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca390b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 0x7f8c9c148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca390b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8c9c152210 0x7f8c9c156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca390b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c0ce0d0 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca390b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c0c58a0 con 0x7f8c9c0c5490 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca390b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f8c9c0c7ff0 con 0x7f8c9c152210 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca0ea6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 0x7f8c9c148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca0ea6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 0x7f8c9c148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51836/0 (socket says 172.21.15.117:51836) 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca0ea6640 1 -- 172.21.15.117:0/2443656714 learned_addr learned my addr 172.21.15.117:0/2443656714 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca1ea8640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8c9c152210 0x7f8c9c156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.314+0000 7f8ca16a7640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c0c5490 0x7f8c9c14a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca0ea6640 1 -- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8c9c152210 msgr2=0x7f8c9c156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca0ea6640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8c9c152210 0x7f8c9c156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca0ea6640 1 -- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c0c5490 msgr2=0x7f8c9c14a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca0ea6640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c0c5490 0x7f8c9c14a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca0ea6640 1 -- 172.21.15.117:0/2443656714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c0cb820 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca1ea8640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8c9c152210 0x7f8c9c156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca0ea6640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 0x7f8c9c148250 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8c980098f0 tx=0x7f8c98005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98031c60 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f8c98004730 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98004b80 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9c149110 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.315+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9c1496d0 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.316+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c9c0c8d00 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.317+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8c98004d20 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.318+0000 7f8c8dffb640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8c80078b90 0x7f8c8007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.318+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f8c980bc5a0 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.318+0000 7f8ca16a7640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8c80078b90 0x7f8c8007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.318+0000 7f8ca16a7640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8c80078b90 0x7f8c8007b0a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8c900015e0 tx=0x7f8c90009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:21.323 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.322+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8c9803f030 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.537+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c9c0c4750 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.537+0000 7f8c8dffb640 1 -- 172.21.15.117:0/2443656714 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6894 (secure 0 0 0) 0x7f8c980845a0 con 0x7f8c9c0c67a0 2024-09-17T05:37:21.538 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:37:21.538 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":44,"btime":"2024-09-17T05:35:55:610284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4337,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/3540182507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":3540182507},{"type":"v1","addr":"172.21.15.119:6837","nonce":3540182507}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4416,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2497891489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2497891489},{"type":"v1","addr":"172.21.15.117:6833","nonce":2497891489}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5084,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2865340338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2865340338},{"type":"v1","addr":"172.21.15.117:6835","nonce":2865340338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":5141,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/714171389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":714171389},{"type":"v1","addr":"172.21.15.119:6839","nonce":714171389}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":5263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2852680530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2852680530},{"type":"v1","addr":"172.21.15.119:6835","nonce":2852680530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T05:27:26.003433+0000","modified":"2024-09-17T05:35:45.569656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5457},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5457":{"gid":5457,"name":"e","rank":0,"incarnation":40,"state":"up:active","state_seq":20,"addr":"172.21.15.117:6837/1696536401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":1696536401},{"type":"v1","addr":"172.21.15.117:6837","nonce":1696536401}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5457,"qdb_cluster":[5457]},"id":1}]} 2024-09-17T05:37:21.538 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 44 2024-09-17T05:37:21.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8c80078b90 msgr2=0x7f8c8007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8c80078b90 0x7f8c8007b0a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8c900015e0 tx=0x7f8c90009040 comp rx=0 tx=0).stop 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 msgr2=0x7f8c9c148250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 0x7f8c9c148250 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8c980098f0 tx=0x7f8c98005360 comp rx=0 tx=0).stop 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 shutdown_connections 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f8c80078b90 0x7f8c8007b0a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8c9c152210 0x7f8c9c156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c9c0c67a0 0x7f8c9c148250 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 --2- 172.21.15.117:0/2443656714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c9c0c5490 0x7f8c9c14a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.539+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 >> 172.21.15.117:0/2443656714 conn(0x7f8c9c06e590 msgr2=0x7f8c9c14f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.540+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 shutdown_connections 2024-09-17T05:37:21.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.540+0000 7f8ca390b640 1 -- 172.21.15.117:0/2443656714 wait complete. 2024-09-17T05:37:21.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:37:21.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 -- 172.21.15.117:0/1140163667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 msgr2=0x7f90300c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1140163667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f90300c9730 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9030074aa0 tx=0x7f902802a9c0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 -- 172.21.15.117:0/1140163667 shutdown_connections 2024-09-17T05:37:21.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1140163667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f90300c9730 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1140163667 >> v2:172.21.15.119:3300/0 conn(0x7f90300cad30 0x7f90300c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 -- 172.21.15.117:0/1140163667 >> 172.21.15.117:0/1140163667 conn(0x7f9030076430 msgr2=0x7f9030072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:21.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 -- 172.21.15.117:0/1140163667 shutdown_connections 2024-09-17T05:37:21.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.676+0000 7f9036cf2640 1 -- 172.21.15.117:0/1140163667 wait complete. 2024-09-17T05:37:21.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.679+0000 7f9036cf2640 1 Processor -- start 2024-09-17T05:37:21.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.679+0000 7f9036cf2640 1 -- start start 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9036cf2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90300cad30 0x7f903014f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9036cf2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9030153260 0x7f903006f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9036cf2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f903013ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9036cf2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f90300cbc30 con 0x7f903015d220 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9036cf2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f903015cd00 con 0x7f90300cad30 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9036cf2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f90300cb230 con 0x7f9030153260 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f903013ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f903013ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51852/0 (socket says 172.21.15.117:51852) 2024-09-17T05:37:21.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 -- 172.21.15.117:0/1389308176 learned_addr learned my addr 172.21.15.117:0/1389308176 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9034a8e640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90300cad30 0x7f903014f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f902ffff640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9030153260 0x7f903006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 -- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9030153260 msgr2=0x7f903006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9030153260 0x7f903006f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 -- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90300cad30 msgr2=0x7f903014f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90300cad30 0x7f903014f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 -- 172.21.15.117:0/1389308176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903006fca0 con 0x7f903015d220 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f9034a8e640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90300cad30 0x7f903014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f903528f640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f903013ff70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9028009d20 tx=0x7f9028030f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.681+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028046070 con 0x7f903015d220 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.682+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f9028039a80 con 0x7f903015d220 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.682+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903013f690 con 0x7f903015d220 2024-09-17T05:37:21.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.682+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028033070 con 0x7f903015d220 2024-09-17T05:37:21.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.682+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f903014d530 con 0x7f903015d220 2024-09-17T05:37:21.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.682+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f903006fe70 con 0x7f903015d220 2024-09-17T05:37:21.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.683+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9028049070 con 0x7f903015d220 2024-09-17T05:37:21.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.684+0000 7f902d7fa640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9014078b90 0x7f901407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:21.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.684+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f90280bc060 con 0x7f903015d220 2024-09-17T05:37:21.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.684+0000 7f9034a8e640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9014078b90 0x7f901407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:21.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.685+0000 7f9034a8e640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9014078b90 0x7f901407b0a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f901c00ab40 tx=0x7f901c00a520 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:21.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.688+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f902803e070 con 0x7f903015d220 2024-09-17T05:37:21.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.873+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90300c37c0 con 0x7f903015d220 2024-09-17T05:37:21.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.874+0000 7f902d7fa640 1 -- 172.21.15.117:0/1389308176 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+19292 (secure 0 0 0) 0x7f90280840d0 con 0x7f903015d220 2024-09-17T05:37:21.875 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-17T05:37:21.875 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":31,"fsid":"8527add3-c46f-4f7f-86e8-5cd706a0043e","created":"2024-09-17T05:27:01.018887+0000","modified":"2024-09-17T05:35:25.910864+0000","last_up_change":"2024-09-17T05:27:14.119421+0000","last_in_change":"2024-09-17T05:27:04.856691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T05:27:13.363384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T05:27:23.869446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T05:27:24.963547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-17T05:27:28.659816+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"83a189b8-5217-4789-a097-5a1555923f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6817","nonce":2282130530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6819","nonce":2282130530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6823","nonce":2282130530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":2282130530},{"type":"v1","addr":"172.21.15.117:6821","nonce":2282130530}]},"public_addr":"172.21.15.117:6817/2282130530","cluster_addr":"172.21.15.117:6819/2282130530","heartbeat_back_addr":"172.21.15.117:6823/2282130530","heartbeat_front_addr":"172.21.15.117:6821/2282130530","state":["exists","up"]},{"osd":1,"uuid":"fab753eb-635a-4a3a-b9aa-323ee34b0b92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6809","nonce":2042501374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6811","nonce":2042501374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6815","nonce":2042501374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2042501374},{"type":"v1","addr":"172.21.15.117:6813","nonce":2042501374}]},"public_addr":"172.21.15.117:6809/2042501374","cluster_addr":"172.21.15.117:6811/2042501374","heartbeat_back_addr":"172.21.15.117:6815/2042501374","heartbeat_front_addr":"172.21.15.117:6813/2042501374","state":["exists","up"]},{"osd":2,"uuid":"5c07e6d8-bcb2-481f-82ab-8500bfbf7719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6801","nonce":2702715806}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6803","nonce":2702715806}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6807","nonce":2702715806}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2702715806},{"type":"v1","addr":"172.21.15.117:6805","nonce":2702715806}]},"public_addr":"172.21.15.117:6801/2702715806","cluster_addr":"172.21.15.117:6803/2702715806","heartbeat_back_addr":"172.21.15.117:6807/2702715806","heartbeat_front_addr":"172.21.15.117:6805/2702715806","state":["exists","up"]},{"osd":3,"uuid":"50957aa1-aed2-438e-8f96-6c81bd139df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6825","nonce":1231670324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6827","nonce":1231670324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6831","nonce":1231670324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":1231670324},{"type":"v1","addr":"172.21.15.117:6829","nonce":1231670324}]},"public_addr":"172.21.15.117:6825/1231670324","cluster_addr":"172.21.15.117:6827/1231670324","heartbeat_back_addr":"172.21.15.117:6831/1231670324","heartbeat_front_addr":"172.21.15.117:6829/1231670324","state":["exists","up"]},{"osd":4,"uuid":"acec14cf-20ec-49de-afeb-8b1fa3e3e602","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6811","nonce":4200593796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6813","nonce":4200593796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6817","nonce":4200593796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":4200593796},{"type":"v1","addr":"172.21.15.119:6815","nonce":4200593796}]},"public_addr":"172.21.15.119:6811/4200593796","cluster_addr":"172.21.15.119:6813/4200593796","heartbeat_back_addr":"172.21.15.119:6817/4200593796","heartbeat_front_addr":"172.21.15.119:6815/4200593796","state":["exists","up"]},{"osd":5,"uuid":"37bb0708-41d4-4ce1-98bb-403acf42b745","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6803","nonce":2281253283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6805","nonce":2281253283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6809","nonce":2281253283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2281253283},{"type":"v1","addr":"172.21.15.119:6807","nonce":2281253283}]},"public_addr":"172.21.15.119:6803/2281253283","cluster_addr":"172.21.15.119:6805/2281253283","heartbeat_back_addr":"172.21.15.119:6809/2281253283","heartbeat_front_addr":"172.21.15.119:6807/2281253283","state":["exists","up"]},{"osd":6,"uuid":"6c2aae50-c4c8-4f33-85ff-ebf6ebf40877","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6819","nonce":2947255794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6821","nonce":2947255794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6825","nonce":2947255794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2947255794},{"type":"v1","addr":"172.21.15.119:6823","nonce":2947255794}]},"public_addr":"172.21.15.119:6819/2947255794","cluster_addr":"172.21.15.119:6821/2947255794","heartbeat_back_addr":"172.21.15.119:6825/2947255794","heartbeat_front_addr":"172.21.15.119:6823/2947255794","state":["exists","up"]},{"osd":7,"uuid":"693ac895-4b40-4737-ba27-8ce2ef7743b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6827","nonce":2580827441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6829","nonce":2580827441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6833","nonce":2580827441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":2580827441},{"type":"v1","addr":"172.21.15.119:6831","nonce":2580827441}]},"public_addr":"172.21.15.119:6827/2580827441","cluster_addr":"172.21.15.119:6829/2580827441","heartbeat_back_addr":"172.21.15.119:6833/2580827441","heartbeat_front_addr":"172.21.15.119:6831/2580827441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.874652+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.402153+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.235910+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.471116+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.173388+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:10.667537+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.133754+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T05:27:11.178981+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3f","mappings":[{"from":0,"to":7}]},{"pgid":"4.f","mappings":[{"from":7,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6839/2500745218":"2024-09-18T05:35:25.910809+0000","172.21.15.119:6838/2500745218":"2024-09-18T05:35:25.910809+0000","172.21.15.119:6835/2224210538":"2024-09-18T05:33:15.875991+0000","172.21.15.119:6834/2224210538":"2024-09-18T05:33:15.875991+0000","172.21.15.117:6835/235108685":"2024-09-18T05:31:45.857073+0000","172.21.15.117:6834/235108685":"2024-09-18T05:31:45.857073+0000","172.21.15.117:6837/242246295":"2024-09-18T05:29:10.830912+0000","172.21.15.117:6836/242246295":"2024-09-18T05:29:10.830912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:37:21.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.876+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9014078b90 msgr2=0x7f901407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9014078b90 0x7f901407b0a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f901c00ab40 tx=0x7f901c00a520 comp rx=0 tx=0).stop 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 msgr2=0x7f903013ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f903013ff70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9028009d20 tx=0x7f9028030f00 comp rx=0 tx=0).stop 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 shutdown_connections 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f9014078b90 0x7f901407b0a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f903015d220 0x7f903013ff70 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9030153260 0x7f903006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 --2- 172.21.15.117:0/1389308176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90300cad30 0x7f903014f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 >> 172.21.15.117:0/1389308176 conn(0x7f9030076430 msgr2=0x7f9030072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 shutdown_connections 2024-09-17T05:37:21.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:21.877+0000 7f9036cf2640 1 -- 172.21.15.117:0/1389308176 wait complete. 2024-09-17T05:37:21.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T05:37:22.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1992239385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60073dd0 msgr2=0x7f3a60152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1992239385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60073dd0 0x7f3a60152f20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005fd0 tx=0x7f3a5002a580 comp rx=0 tx=0).stop 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1992239385 shutdown_connections 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1992239385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60073dd0 0x7f3a60152f20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1992239385 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a600c8a00 0x7f3a600c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1992239385 >> v2:172.21.15.119:3300/0 conn(0x7f3a60159110 0x7f3a60074e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1992239385 >> 172.21.15.117:0/1992239385 conn(0x7f3a6006e590 msgr2=0x7f3a600cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1992239385 shutdown_connections 2024-09-17T05:37:22.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.015+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1992239385 wait complete. 2024-09-17T05:37:22.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.017+0000 7f3a67c0e640 1 Processor -- start 2024-09-17T05:37:22.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.017+0000 7f3a67c0e640 1 -- start start 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a67c0e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a60073dd0 0x7f3a601eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a67c0e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a600c8a00 0x7f3a600ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a67c0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 0x7f3a600cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a67c0e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a60071e40 con 0x7f3a60159110 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a67c0e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a60074930 con 0x7f3a600c8a00 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a67c0e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7f3a600c9dc0 con 0x7f3a60073dd0 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a661ab640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 0x7f3a600cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a661ab640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 0x7f3a600cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51878/0 (socket says 172.21.15.117:51878) 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.018+0000 7f3a661ab640 1 -- 172.21.15.117:0/1748234956 learned_addr learned my addr 172.21.15.117:0/1748234956 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a651a9640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a600c8a00 0x7f3a600ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a659aa640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a60073dd0 0x7f3a601eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a661ab640 1 -- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a60073dd0 msgr2=0x7f3a601eff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a661ab640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a60073dd0 0x7f3a601eff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a661ab640 1 -- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a600c8a00 msgr2=0x7f3a600ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a661ab640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a600c8a00 0x7f3a600ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a661ab640 1 -- 172.21.15.117:0/1748234956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a601e0e90 con 0x7f3a60159110 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a659aa640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a60073dd0 0x7f3a601eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a661ab640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 0x7f3a600cd2d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800e8e0 tx=0x7f3a5800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a58004d20 con 0x7f3a60159110 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f3a5800ca60 con 0x7f3a60159110 2024-09-17T05:37:22.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a601f0440 con 0x7f3a60159110 2024-09-17T05:37:22.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a58005020 con 0x7f3a60159110 2024-09-17T05:37:22.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.019+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a601ef700 con 0x7f3a60159110 2024-09-17T05:37:22.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.020+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a601e13b0 con 0x7f3a60159110 2024-09-17T05:37:22.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.022+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a580051c0 con 0x7f3a60159110 2024-09-17T05:37:22.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.022+0000 7f3a567fc640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3a44078b90 0x7f3a4407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.022+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6934+0+0 (secure 0 0 0) 0x7f3a5809bc10 con 0x7f3a60159110 2024-09-17T05:37:22.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.022+0000 7f3a659aa640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3a44078b90 0x7f3a4407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.023+0000 7f3a659aa640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3a44078b90 0x7f3a4407b0a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c005fd0 tx=0x7f3a5c00f360 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:22.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.027+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a601e13b0 con 0x7f3a60159110 2024-09-17T05:37:22.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.244+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3a601ef4f0 con 0x7f3a60159110 2024-09-17T05:37:22.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.246+0000 7f3a567fc640 1 -- 172.21.15.117:0/1748234956 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7f3a58016030 con 0x7f3a60159110 2024-09-17T05:37:22.247 INFO:teuthology.orchestra.run.smithi117.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T05:37:22.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3a44078b90 msgr2=0x7f3a4407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3a44078b90 0x7f3a4407b0a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c005fd0 tx=0x7f3a5c00f360 comp rx=0 tx=0).stop 2024-09-17T05:37:22.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 msgr2=0x7f3a600cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 0x7f3a600cd2d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800e8e0 tx=0x7f3a5800eda0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 shutdown_connections 2024-09-17T05:37:22.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7f3a44078b90 0x7f3a4407b0a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a60159110 0x7f3a600cd2d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a600c8a00 0x7f3a600ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 --2- 172.21.15.117:0/1748234956 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3a60073dd0 0x7f3a601eff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 >> 172.21.15.117:0/1748234956 conn(0x7f3a6006e590 msgr2=0x7f3a600ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:22.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.251+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 shutdown_connections 2024-09-17T05:37:22.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.252+0000 7f3a67c0e640 1 -- 172.21.15.117:0/1748234956 wait complete. 2024-09-17T05:37:22.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T05:37:22.325 INFO:tasks.ceph.mds.e.smithi117.stdout:starting mds.e at 2024-09-17T05:37:22.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3639220061 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 msgr2=0x7fbe68003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3639220061 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 0x7fbe68003ec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c006fd0 tx=0x7fbe68005830 comp rx=0 tx=0).stop 2024-09-17T05:37:22.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3639220061 shutdown_connections 2024-09-17T05:37:22.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3639220061 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 0x7fbe68003ec0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3639220061 >> 172.21.15.117:0/3639220061 conn(0x7fbe78076430 msgr2=0x7fbe78072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:22.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3639220061 shutdown_connections 2024-09-17T05:37:22.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.385+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3639220061 wait complete. 2024-09-17T05:37:22.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.387+0000 7fbe7f1ed640 1 Processor -- start 2024-09-17T05:37:22.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.387+0000 7fbe7f1ed640 1 -- start start 2024-09-17T05:37:22.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.388+0000 7fbe7f1ed640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 0x7fbe78141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.388+0000 7fbe7f1ed640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7814d6d0 0x7fbe7815be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.388+0000 7fbe7f1ed640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 0x7fbe7815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.388+0000 7fbe7f1ed640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbe780cb230 con 0x7fbe7815ae40 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.388+0000 7fbe7f1ed640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbe780cbc30 con 0x7fbe7814d6d0 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.388+0000 7fbe7f1ed640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fbe780cb3b0 con 0x7fbe68003a50 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 0x7fbe7815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 0x7fbe7815b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51896/0 (socket says 172.21.15.117:51896) 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 -- 172.21.15.117:0/3523694942 learned_addr learned my addr 172.21.15.117:0/3523694942 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe77fff640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7814d6d0 0x7fbe7815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7cf89640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 0x7fbe78141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 -- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 msgr2=0x7fbe78141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 0x7fbe78141490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 -- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7814d6d0 msgr2=0x7fbe7815be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:22.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7814d6d0 0x7fbe7815be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 -- 172.21.15.117:0/3523694942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe781419d0 con 0x7fbe7815ae40 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7d78a640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 0x7fbe7815b250 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbe70005fd0 tx=0x7fbe7000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70004b20 con 0x7fbe7815ae40 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fbe70004dd0 con 0x7fbe7815ae40 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe780b3dc0 con 0x7fbe7815ae40 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe70003bd0 con 0x7fbe7815ae40 2024-09-17T05:37:22.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.389+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7815a4c0 con 0x7fbe7815ae40 2024-09-17T05:37:22.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.390+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe780b41e0 con 0x7fbe7815ae40 2024-09-17T05:37:22.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.391+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbe70004030 con 0x7fbe7815ae40 2024-09-17T05:37:22.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.392+0000 7fbe757fa640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbe5c078b90 0x7fbe5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:22.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.392+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7020+0+0 (secure 0 0 0) 0x7fbe7009bba0 con 0x7fbe7815ae40 2024-09-17T05:37:22.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.392+0000 7fbe7cf89640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbe5c078b90 0x7fbe5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:22.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.392+0000 7fbe7cf89640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbe5c078b90 0x7fbe5c07b0a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c033000 tx=0x7fbe780cb840 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:22.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.396+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbe781419d0 con 0x7fbe7815ae40 2024-09-17T05:37:22.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:22.607+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbe7815d220 con 0x7fbe7815ae40 2024-09-17T05:37:23.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.248+0000 7fbe757fa640 1 -- 172.21.15.117:0/3523694942 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) ==== 104+0+0 (secure 0 0 0) 0x7fbe70063510 con 0x7fbe7815ae40 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbe5c078b90 msgr2=0x7fbe5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbe5c078b90 0x7fbe5c07b0a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c033000 tx=0x7fbe780cb840 comp rx=0 tx=0).stop 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 msgr2=0x7fbe7815b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 0x7fbe7815b250 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbe70005fd0 tx=0x7fbe7000f3c0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 shutdown_connections 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbe5c078b90 0x7fbe5c07b0a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe7815ae40 0x7fbe7815b250 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7814d6d0 0x7fbe7815be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 --2- 172.21.15.117:0/3523694942 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbe68003a50 0x7fbe78141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 >> 172.21.15.117:0/3523694942 conn(0x7fbe78076430 msgr2=0x7fbe78157e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:23.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 shutdown_connections 2024-09-17T05:37:23.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.250+0000 7fbe7f1ed640 1 -- 172.21.15.117:0/3523694942 wait complete. 2024-09-17T05:37:23.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T05:37:23.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 -- 172.21.15.117:0/1204537806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500c7800 msgr2=0x7fd050151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:23.384 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 --2- 172.21.15.117:0/1204537806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500c7800 0x7fd050151fe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd050074f10 tx=0x7fd04002a5d0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 -- 172.21.15.117:0/1204537806 shutdown_connections 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 --2- 172.21.15.117:0/1204537806 >> v2:172.21.15.119:3300/0 conn(0x7fd0500ce040 0x7fd0500c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 --2- 172.21.15.117:0/1204537806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500c7800 0x7fd050151fe0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 -- 172.21.15.117:0/1204537806 >> 172.21.15.117:0/1204537806 conn(0x7fd05006e570 msgr2=0x7fd0500cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 -- 172.21.15.117:0/1204537806 shutdown_connections 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.383+0000 7fd058263640 1 -- 172.21.15.117:0/1204537806 wait complete. 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.385+0000 7fd058263640 1 Processor -- start 2024-09-17T05:37:23.385 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.385+0000 7fd058263640 1 -- start start 2024-09-17T05:37:23.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd058263640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd050071930 0x7fd0501f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:23.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd058263640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0500c7800 0x7fd050150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:23.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd058263640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 0x7fd050142090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:23.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd058263640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500c9680 con 0x7fd0500ce040 2024-09-17T05:37:23.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd058263640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500c5170 con 0x7fd0500c7800 2024-09-17T05:37:23.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd058263640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fd0500c9800 con 0x7fd050071930 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd056800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 0x7fd050142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd056800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 0x7fd050142090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51912/0 (socket says 172.21.15.117:51912) 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.386+0000 7fd056800640 1 -- 172.21.15.117:0/377654849 learned_addr learned my addr 172.21.15.117:0/377654849 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd055fff640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd050071930 0x7fd0501f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd0557fe640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0500c7800 0x7fd050150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd056800640 1 -- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd050071930 msgr2=0x7fd0501f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd056800640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd050071930 0x7fd0501f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd056800640 1 -- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0500c7800 msgr2=0x7fd050150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd056800640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0500c7800 0x7fd050150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd056800640 1 -- 172.21.15.117:0/377654849 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0501f3b40 con 0x7fd0500ce040 2024-09-17T05:37:23.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd056800640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 0x7fd050142090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c760 tx=0x7fd04800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:23.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048003b70 con 0x7fd0500ce040 2024-09-17T05:37:23.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fd048003d10 con 0x7fd0500ce040 2024-09-17T05:37:23.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05014d7e0 con 0x7fd0500ce040 2024-09-17T05:37:23.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0480052d0 con 0x7fd0500ce040 2024-09-17T05:37:23.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.387+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0501f2e00 con 0x7fd0500ce040 2024-09-17T05:37:23.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.389+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0500c6850 con 0x7fd0500ce040 2024-09-17T05:37:23.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.389+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd048005470 con 0x7fd0500ce040 2024-09-17T05:37:23.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.389+0000 7fd0467fc640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd034078b90 0x7fd03407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:23.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.390+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7020+0+0 (secure 0 0 0) 0x7fd04809b020 con 0x7fd0500ce040 2024-09-17T05:37:23.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.390+0000 7fd055fff640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd034078b90 0x7fd03407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:23.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.390+0000 7fd055fff640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd034078b90 0x7fd03407b0a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd040034000 tx=0x7fd0400046c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:23.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.394+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0500c6850 con 0x7fd0500ce040 2024-09-17T05:37:23.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:23.577+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd050151650 con 0x7fd0500ce040 2024-09-17T05:37:24.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.253+0000 7fd0467fc640 1 -- 172.21.15.117:0/377654849 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v33) ==== 181+0+0 (secure 0 0 0) 0x7fd048062980 con 0x7fd0500ce040 2024-09-17T05:37:24.254 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_metadata' removed 2024-09-17T05:37:24.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd034078b90 msgr2=0x7fd03407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:24.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd034078b90 0x7fd03407b0a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd040034000 tx=0x7fd0400046c0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 msgr2=0x7fd050142090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:24.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 0x7fd050142090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c760 tx=0x7fd04800cb60 comp rx=0 tx=0).stop 2024-09-17T05:37:24.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 shutdown_connections 2024-09-17T05:37:24.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fd034078b90 0x7fd03407b0a0 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd040034000 tx=0x7fd0400046c0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0500ce040 0x7fd050142090 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0500c7800 0x7fd050150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 --2- 172.21.15.117:0/377654849 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd050071930 0x7fd0501f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.257+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 >> 172.21.15.117:0/377654849 conn(0x7fd05006e570 msgr2=0x7fd0500c9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:24.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.259+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 shutdown_connections 2024-09-17T05:37:24.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.259+0000 7fd058263640 1 -- 172.21.15.117:0/377654849 wait complete. 2024-09-17T05:37:24.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T05:37:24.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 -- 172.21.15.117:0/820008121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c4eb0 msgr2=0x7fbc800c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:24.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 --2- 172.21.15.117:0/820008121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c4eb0 0x7fbc800c52c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009830 tx=0x7fbc6802a510 comp rx=0 tx=0).stop 2024-09-17T05:37:24.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 -- 172.21.15.117:0/820008121 shutdown_connections 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 --2- 172.21.15.117:0/820008121 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c9040 0x7fbc800c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 --2- 172.21.15.117:0/820008121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c4eb0 0x7fbc800c52c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 -- 172.21.15.117:0/820008121 >> 172.21.15.117:0/820008121 conn(0x7fbc800740e0 msgr2=0x7fbc8006deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 -- 172.21.15.117:0/820008121 shutdown_connections 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.395+0000 7fbc85969640 1 -- 172.21.15.117:0/820008121 wait complete. 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.397+0000 7fbc85969640 1 Processor -- start 2024-09-17T05:37:24.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.397+0000 7fbc85969640 1 -- start start 2024-09-17T05:37:24.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc85969640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c4eb0 0x7fbc800cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:24.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc85969640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 0x7fbc80150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:24.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc85969640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc80152120 0x7fbc8013dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc85969640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbc800c9990 con 0x7fbc800c9040 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc85969640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbc80152530 con 0x7fbc80152120 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc85969640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fbc8015c3b0 con 0x7fbc800c4eb0 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc7e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 0x7fbc80150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc7e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 0x7fbc80150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42126/0 (socket says 172.21.15.117:42126) 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.398+0000 7fbc7e7fc640 1 -- 172.21.15.117:0/4063628087 learned_addr learned my addr 172.21.15.117:0/4063628087 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7effd640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c4eb0 0x7fbc800cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7f7fe640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc80152120 0x7fbc8013dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7e7fc640 1 -- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c4eb0 msgr2=0x7fbc800cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7e7fc640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c4eb0 0x7fbc800cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7e7fc640 1 -- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc80152120 msgr2=0x7fbc8013dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:24.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7e7fc640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc80152120 0x7fbc8013dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7e7fc640 1 -- 172.21.15.117:0/4063628087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc80143a20 con 0x7fbc800c9040 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7effd640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c4eb0 0x7fbc800cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc7e7fc640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 0x7fbc80150d30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbc6802ab00 tx=0x7fbc68005430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc68005620 con 0x7fbc800c9040 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fbc68004730 con 0x7fbc800c9040 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc8015c110 con 0x7fbc800c9040 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc68004b80 con 0x7fbc800c9040 2024-09-17T05:37:24.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.399+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc80143630 con 0x7fbc800c9040 2024-09-17T05:37:24.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.400+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc800c7870 con 0x7fbc800c9040 2024-09-17T05:37:24.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.401+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbc68031770 con 0x7fbc800c9040 2024-09-17T05:37:24.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.401+0000 7fbc77fff640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbc60078b90 0x7fbc6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:24.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.402+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 6582+0+0 (secure 0 0 0) 0x7fbc680bc400 con 0x7fbc800c9040 2024-09-17T05:37:24.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.402+0000 7fbc7effd640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbc60078b90 0x7fbc6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:24.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.402+0000 7fbc7effd640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbc60078b90 0x7fbc6007b0a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbc700016a0 tx=0x7fbc70009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:24.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.406+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbc6803f030 con 0x7fbc800c9040 2024-09-17T05:37:24.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:24.595+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbc801559a0 con 0x7fbc800c9040 2024-09-17T05:37:25.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.259+0000 7fbc77fff640 1 -- 172.21.15.117:0/4063628087 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v34) ==== 169+0+0 (secure 0 0 0) 0x7fbc68083f10 con 0x7fbc800c9040 2024-09-17T05:37:25.260 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data' removed 2024-09-17T05:37:25.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.261+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbc60078b90 msgr2=0x7fbc6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:25.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.261+0000 7fbc85969640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbc60078b90 0x7fbc6007b0a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbc700016a0 tx=0x7fbc70009040 comp rx=0 tx=0).stop 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.261+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 msgr2=0x7fbc80150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.261+0000 7fbc85969640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 0x7fbc80150d30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbc6802ab00 tx=0x7fbc68005430 comp rx=0 tx=0).stop 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 shutdown_connections 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fbc60078b90 0x7fbc6007b0a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc80152120 0x7fbc8013dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc800c9040 0x7fbc80150d30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 --2- 172.21.15.117:0/4063628087 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbc800c4eb0 0x7fbc800cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 >> 172.21.15.117:0/4063628087 conn(0x7fbc800740e0 msgr2=0x7fbc80071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 shutdown_connections 2024-09-17T05:37:25.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.262+0000 7fbc85969640 1 -- 172.21.15.117:0/4063628087 wait complete. 2024-09-17T05:37:25.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 -- 172.21.15.117:0/51300169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc0cf250 msgr2=0x7fccfc153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 --2- 172.21.15.117:0/51300169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc0cf250 0x7fccfc153120 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fccec005fd0 tx=0x7fccec02a670 comp rx=0 tx=0).stop 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 -- 172.21.15.117:0/51300169 shutdown_connections 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 --2- 172.21.15.117:0/51300169 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 0x7fccfc072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 --2- 172.21.15.117:0/51300169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc0cf250 0x7fccfc153120 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 -- 172.21.15.117:0/51300169 >> 172.21.15.117:0/51300169 conn(0x7fccfc0722b0 msgr2=0x7fccfc075e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 -- 172.21.15.117:0/51300169 shutdown_connections 2024-09-17T05:37:25.392 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.391+0000 7fcd03da1640 1 -- 172.21.15.117:0/51300169 wait complete. 2024-09-17T05:37:25.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.392+0000 7fcd03da1640 1 Processor -- start 2024-09-17T05:37:25.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.393+0000 7fcd03da1640 1 -- start start 2024-09-17T05:37:25.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd03da1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 0x7fccfc153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:25.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd03da1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccfc0cf250 0x7fccfc147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:25.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd03da1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 0x7fccfc157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:25.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd03da1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc0c3cd0 con 0x7fccfc072bd0 2024-09-17T05:37:25.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd03da1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc0c39f0 con 0x7fccfc0cf250 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd03da1640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc0c3fb0 con 0x7fccfc159810 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 0x7fccfc153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 0x7fccfc153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42148/0 (socket says 172.21.15.117:42148) 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 -- 172.21.15.117:0/4231185602 learned_addr learned my addr 172.21.15.117:0/4231185602 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd0133c640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccfc0cf250 0x7fccfc147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd0233e640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 0x7fccfc157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 -- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 msgr2=0x7fccfc157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 0x7fccfc157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 -- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccfc0cf250 msgr2=0x7fccfc147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:25.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccfc0cf250 0x7fccfc147410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 -- 172.21.15.117:0/4231185602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc1540e0 con 0x7fccfc072bd0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd0233e640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 0x7fccfc157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fcd01b3d640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 0x7fccfc153ba0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fccec034040 tx=0x7fccec003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.394+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec03e030 con 0x7fccfc072bd0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.395+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fccec003e40 con 0x7fccfc072bd0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.395+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc1490c0 con 0x7fccfc072bd0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.395+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec035040 con 0x7fccfc072bd0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.395+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc149680 con 0x7fccfc072bd0 2024-09-17T05:37:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.396+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc0c4e90 con 0x7fccfc072bd0 2024-09-17T05:37:25.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.396+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fccec041030 con 0x7fccfc072bd0 2024-09-17T05:37:25.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.397+0000 7fccf27fc640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcce0078bc0 0x7fcce007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:37:25.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.397+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6139+0+0 (secure 0 0 0) 0x7fccec0bb4f0 con 0x7fccfc072bd0 2024-09-17T05:37:25.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.397+0000 7fcd0133c640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcce0078bc0 0x7fcce007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:37:25.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.398+0000 7fcd0133c640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcce0078bc0 0x7fcce007b0d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fccfc1e2070 tx=0x7fccf8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:37:25.402 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.401+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fccfc0c4e90 con 0x7fccfc072bd0 2024-09-17T05:37:25.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:25.588+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fccfc0c5610 con 0x7fccfc072bd0 2024-09-17T05:37:26.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.261+0000 7fccf27fc640 1 -- 172.21.15.117:0/4231185602 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v35) ==== 178+0+0 (secure 0 0 0) 0x7fccec0831c0 con 0x7fccfc072bd0 2024-09-17T05:37:26.262 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data_ec' removed 2024-09-17T05:37:26.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcce0078bc0 msgr2=0x7fcce007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:26.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcce0078bc0 0x7fcce007b0d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fccfc1e2070 tx=0x7fccf8009040 comp rx=0 tx=0).stop 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 msgr2=0x7fccfc153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 0x7fccfc153ba0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fccec034040 tx=0x7fccec003910 comp rx=0 tx=0).stop 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 shutdown_connections 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:6800/61506279,v1:172.21.15.119:6801/61506279] conn(0x7fcce0078bc0 0x7fcce007b0d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fccfc159810 0x7fccfc157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccfc0cf250 0x7fccfc147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 --2- 172.21.15.117:0/4231185602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccfc072bd0 0x7fccfc153ba0 secure :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fccec034040 tx=0x7fccec003910 comp rx=0 tx=0).stop 2024-09-17T05:37:26.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.265+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 >> 172.21.15.117:0/4231185602 conn(0x7fccfc0722b0 msgr2=0x7fccfc1522b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:37:26.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.266+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 shutdown_connections 2024-09-17T05:37:26.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-17T05:37:26.266+0000 7fcd03da1640 1 -- 172.21.15.117:0/4231185602 wait complete. 2024-09-17T05:37:26.284 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T05:37:26.285 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:37:26.285 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.297 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:37:26.297 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:37:26.297 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.351 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:37:26.351 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-09-17T05:37:26.351 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.409 INFO:tasks.ceph.mds.e:Stopped 2024-09-17T05:37:26.409 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T05:37:26.409 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.423 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T05:37:26.423 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:37:26.423 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.479 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:37:26.479 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-09-17T05:37:26.479 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.536 INFO:tasks.ceph.mds.f:Stopped 2024-09-17T05:37:26.536 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T05:37:26.536 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T05:37:26.536 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.623 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T05:37:26.624 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T05:37:26.624 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:26.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-09-17T05:37:26.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-17T05:37:26.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-17T05:37:26.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-09-17T05:37:32.725 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T05:37:32.725 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T05:37:32.726 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:32.789 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T05:37:32.789 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T05:37:32.789 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-17T05:37:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e has been restored 2024-09-17T05:37:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-17T05:37:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-17T05:37:32.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f has been restored 2024-09-17T05:37:32.866 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T05:37:32.866 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T05:37:32.866 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:32.953 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T05:37:32.953 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T05:37:32.953 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.024 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T05:37:33.024 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-17T05:37:33.024 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.095 INFO:tasks.ceph.osd.6:Stopped 2024-09-17T05:37:33.095 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-17T05:37:33.095 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.172 INFO:tasks.ceph.osd.7:Stopped 2024-09-17T05:37:33.173 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T05:37:33.173 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-17T05:37:33.173 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.203 INFO:tasks.ceph.mgr.y:Stopped 2024-09-17T05:37:33.204 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T05:37:33.204 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.232 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T05:37:33.232 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T05:37:33.232 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T05:37:33.232 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.255 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T05:37:33.255 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T05:37:33.255 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.283 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T05:37:33.283 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T05:37:33.283 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:37:33.334 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T05:37:33.334 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T05:37:33.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-17T05:37:33.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:37:33.434 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T05:37:35.178 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:37:35.178 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T05:37:35.439 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:37:35.439 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T05:37:35.598 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T05:37:35.598 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T05:37:35.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:37:35.699 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T05:37:36.237 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:37:36.237 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T05:37:36.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:37:36.373 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-17T05:37:36.468 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T05:37:36.468 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-17T05:37:36.561 INFO:tasks.ceph:Archiving mon data... 2024-09-17T05:37:36.562 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/data/mon.a.tgz 2024-09-17T05:37:36.562 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-09-17T05:37:36.577 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.0NXP8jDy7k 2024-09-17T05:37:36.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0NXP8jDy7k 2024-09-17T05:37:36.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.0NXP8jDy7k 2024-09-17T05:37:36.830 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.0NXP8jDy7k is 346KB 2024-09-17T05:37:36.897 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.0NXP8jDy7k 2024-09-17T05:37:36.912 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/data/mon.b.tgz 2024-09-17T05:37:36.912 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-17T05:37:36.930 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.VZPIJdpskK 2024-09-17T05:37:36.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VZPIJdpskK 2024-09-17T05:37:37.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.VZPIJdpskK 2024-09-17T05:37:37.191 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.VZPIJdpskK is 356KB 2024-09-17T05:37:37.259 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.VZPIJdpskK 2024-09-17T05:37:37.274 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/data/mon.c.tgz 2024-09-17T05:37:37.274 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-17T05:37:37.328 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.aBdlMvStTU 2024-09-17T05:37:37.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.aBdlMvStTU 2024-09-17T05:37:37.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.aBdlMvStTU 2024-09-17T05:37:37.541 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.aBdlMvStTU is 356KB 2024-09-17T05:37:37.636 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.aBdlMvStTU 2024-09-17T05:37:37.651 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T05:37:37.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T05:37:37.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T05:37:37.760 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T05:37:37.782 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T05:37:37.783 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T05:37:37.784 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/remote/smithi117/crash 2024-09-17T05:37:37.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T05:37:37.825 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/remote/smithi119/crash 2024-09-17T05:37:37.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T05:37:37.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-17T05:37:37.849 INFO:tasks.ceph:Shutting down logrotate 2024-09-17T05:39:06.267 INFO:teuthology.orchestra.run.smithi117.stderr:error: Compressing program wrote following message to stderr when compressing log /var/log/ceph/ceph-osd.3.log.1: 2024-09-17T05:39:06.268 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: stdin: file size changed while zipping 2024-09-17T05:39:07.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-17T05:39:07.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-17T05:39:07.155 INFO:tasks.ceph:Compressing logs... 2024-09-17T05:39:07.155 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T05:39:07.158 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T05:39:07.181 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52972.log 2024-09-17T05:39:07.182 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T05:39:07.182 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52147.log 2024-09-17T05:39:07.183 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51845.log 2024-09-17T05:39:07.183 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44090.log 2024-09-17T05:39:07.183 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55081.log 2024-09-17T05:39:07.212 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37465.log 2024-09-17T05:39:07.212 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37425.log 2024-09-17T05:39:07.212 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-17T05:39:07.212 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37465.log: /var/log/ceph/ceph-client.admin.37425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37225.log 2024-09-17T05:39:07.212 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-17T05:39:07.212 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37225.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37185.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37345.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37345.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37265.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37305.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-09-17T05:39:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37385.log 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mds.f.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37425.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.37385.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37465.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37185.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37225.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37265.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37345.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37305.log.gz 2024-09-17T05:39:07.214 INFO:teuthology.orchestra.run.smithi119.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37385.log.gz 2024-09-17T05:39:07.295 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55123.log 2024-09-17T05:39:07.295 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54200.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55081.log.gz 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50727.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55123.log.gz 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37405.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54200.log.gz 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45246.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54507.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54372.log 2024-09-17T05:39:07.296 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52930.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54372.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54507.log.gz 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52465.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54372.log.gz 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50685.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43918.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37416.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36684.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53368.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37164.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44976.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52888.log 2024-09-17T05:39:07.297 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51761.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37392.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46146.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50895.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44096.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42378.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53182.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44096.log: /var/log/ceph/ceph-client.admin.42378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46232.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44778.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43792.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.46401.log 2024-09-17T05:39:07.298 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44435.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51803.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52972.log.gz 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37226.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56106.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53368.log.gz 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46763.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.56106.log: /var/log/ceph/ceph-client.admin.44089.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56106.log.gz 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52191.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52423.log 2024-09-17T05:39:07.299 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55585.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51301.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53411.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42336.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55585.log.gz 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42038.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36741.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44099.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42570.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47571.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46317.log 2024-09-17T05:39:07.300 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45289.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41952.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52930.log.gz 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46485.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44436.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52507.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47413.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44774.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44772.log 2024-09-17T05:39:07.301 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43708.log 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47455.log 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr: 77.7% -- replaced with /var/log/ceph/ceph-client.admin.36684.log.gz 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41675.log 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.37226.log.gz 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53266.log 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1%/var/log/ceph/ceph-client.admin.41675.log: -- replaced with /var/log/ceph/ceph-client.admin.37164.log.gz 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45018.log 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37392.log.gz 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55459.log 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53266.log: /var/log/ceph/ceph-client.admin.45018.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.37405.log.gz 2024-09-17T05:39:07.302 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38164.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.37416.log.gz 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53014.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38164.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.55459.log.gz 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52846.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45060.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53014.log: /var/log/ceph/ceph-client.admin.52846.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53266.log.gz 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55753.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36853.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55375.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55753.log.gz 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2024-09-17T05:39:07.303 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.53411.log.gz 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36768.log 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55375.log: /var/log/ceph/ceph-client.admin.45373.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55375.log.gz 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36825.log 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52465.log.gz 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52339.log 2024-09-17T05:39:07.304 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51700.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47140.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46275.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47140.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43295.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37399.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44434.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51217.log 2024-09-17T05:39:07.305 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37404.log 2024-09-17T05:39:07.306 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51217.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42378.log.gz 2024-09-17T05:39:07.306 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42420.log 2024-09-17T05:39:07.306 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52678.log 2024-09-17T05:39:07.306 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42082.log 2024-09-17T05:39:07.306 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54242.log 2024-09-17T05:39:07.306 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51175.log 2024-09-17T05:39:07.307 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50987.log 2024-09-17T05:39:07.307 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54242.log.gz 2024-09-17T05:39:07.307 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53495.log 2024-09-17T05:39:07.307 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54731.log 2024-09-17T05:39:07.307 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53537.log 2024-09-17T05:39:07.308 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.53495.log.gz 2024-09-17T05:39:07.308 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44087.log 2024-09-17T05:39:07.308 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54731.log.gz 2024-09-17T05:39:07.308 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43792.log.gz 2024-09-17T05:39:07.308 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50625.log 2024-09-17T05:39:07.309 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43918.log.gz 2024-09-17T05:39:07.309 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53537.log.gz 2024-09-17T05:39:07.309 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54552.log 2024-09-17T05:39:07.309 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44087.log: 86.9%/var/log/ceph/ceph-client.admin.50625.log: -- replaced with /var/log/ceph/ceph-client.admin.44096.log.gz 2024-09-17T05:39:07.309 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53705.log 2024-09-17T05:39:07.309 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2024-09-17T05:39:07.310 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37387.log 2024-09-17T05:39:07.310 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53705.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.54552.log.gz 2024-09-17T05:39:07.310 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51386.log 2024-09-17T05:39:07.310 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44099.log.gz 2024-09-17T05:39:07.310 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44090.log.gz 87.0% 2024-09-17T05:39:07.310 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53705.log.gz 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44780.log 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56148.log 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55333.log 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.56148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44429.log 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3%/var/log/ceph/ceph-client.admin.55333.log: -- replaced with /var/log/ceph/ceph-client.admin.56148.log.gz 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55877.log 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.55333.log.gz 2024-09-17T05:39:07.311 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52233.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46721.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55877.log.gz 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52762.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51344.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45634.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52591.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55711.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45634.log: /var/log/ceph/ceph-client.admin.52591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43540.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54955.log 2024-09-17T05:39:07.312 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55711.log.gz 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55417.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54955.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42570.log.gz 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54955.log.gz 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44659.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.55417.log.gz 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53832.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.44659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51554.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53453.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53832.log.gz 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41593.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55249.log 2024-09-17T05:39:07.313 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53453.log.gz 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54997.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43237.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55249.log.gz 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44775.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54997.log.gz 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43010.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51029.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52804.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41910.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45162.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52804.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.41910.log: -- /var/log/ceph/ceph-client.admin.53937.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43750.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51971.log 2024-09-17T05:39:07.314 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53937.log.gz 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5/var/log/ceph/ceph-client.admin.43750.log: --verbose -- /var/log/ceph/ceph-client.admin.54150.log 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47182.log 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52633.log 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6%/var/log/ceph/ceph-client.admin.47182.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46146.log.gz 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54150.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45331.log 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46232.log.gz 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46443.log 2024-09-17T05:39:07.315 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52633.log: /var/log/ceph/ceph-client.admin.45331.log: 87.2% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46317.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46275.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46401.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37378.log 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46485.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36797.log 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50895.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37166.log 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51301.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.52888.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37364.log 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36797.log.gz 2024-09-17T05:39:07.316 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51887.log 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37166.log: /var/log/ceph/ceph-client.admin.37364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37234.log 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53663.log 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41910.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41952.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44100.log 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53663.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.56045.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42038.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42082.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54064.log 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2024-09-17T05:39:07.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51259.log 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54064.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51259.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51512.log 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54064.log.gz 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36881.log 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51428.log 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2024-09-17T05:39:07.318 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55039.log 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46847.log 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37371.log 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55039.log.gz 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47048.log 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37371.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.44435.log.gz 2024-09-17T05:39:07.319 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55207.log 2024-09-17T05:39:07.320 INFO:teuthology.orchestra.run.smithi117.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42336.log.gz 2024-09-17T05:39:07.320 INFO:teuthology.orchestra.run.smithi117.stderr: -5/var/log/ceph/ceph-client.admin.47048.log: --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42420.log.gz 2024-09-17T05:39:07.320 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.53747.log 2024-09-17T05:39:07.320 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55207.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51115.log 2024-09-17T05:39:07.320 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44434.log.gz 2024-09-17T05:39:07.320 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.55207.log.gz 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55165.log 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43237.log.gz 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51115.log: /var/log/ceph/ceph-client.admin.53895.log 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44429.log.gz 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53098.log 2024-09-17T05:39:07.321 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55165.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53747.log.gz 2024-09-17T05:39:07.322 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2024-09-17T05:39:07.322 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37012.log 2024-09-17T05:39:07.322 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.322 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-09-17T05:39:07.322 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44772.log.gz 2024-09-17T05:39:07.322 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.55165.log.gz 2024-09-17T05:39:07.323 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44774.log.gz 2024-09-17T05:39:07.323 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44659.log.gz 2024-09-17T05:39:07.323 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-17T05:39:07.323 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53098.log: /var/log/ceph/ceph-client.admin.53895.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44780.log.gz 2024-09-17T05:39:07.323 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44976.log.gz 2024-09-17T05:39:07.323 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44436.log.gz 2024-09-17T05:39:07.324 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5/var/log/ceph/ceph-client.admin.37012.log: --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53895.log.gz 2024-09-17T05:39:07.324 INFO:teuthology.orchestra.run.smithi117.stderr: -- /var/log/ceph/ceph-client.admin.37867.log 2024-09-17T05:39:07.324 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45018.log.gz 2024-09-17T05:39:07.324 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2024-09-17T05:39:07.324 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mgr.y.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45162.log.gz 2024-09-17T05:39:07.324 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44095.log 2024-09-17T05:39:07.325 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45246.log.gz 2024-09-17T05:39:07.325 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43414.log 2024-09-17T05:39:07.325 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50811.log 2024-09-17T05:39:07.325 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55961.log 2024-09-17T05:39:07.325 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-09-17T05:39:07.326 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36964.log 2024-09-17T05:39:07.326 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55961.log.gz 2024-09-17T05:39:07.326 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55795.log 2024-09-17T05:39:07.326 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.36964.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T05:39:07.326 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53621.log 2024-09-17T05:39:07.326 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55795.log.gz 2024-09-17T05:39:07.327 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54414.log 2024-09-17T05:39:07.327 INFO:teuthology.orchestra.run.smithi117.stderr:gzip/var/log/ceph/ceph-client.admin.53621.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47227.log 2024-09-17T05:39:07.327 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54414.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52105.log 2024-09-17T05:39:07.327 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53621.log.gz 2024-09-17T05:39:07.327 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54414.log.gz 2024-09-17T05:39:07.327 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37827.log 2024-09-17T05:39:07.328 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50937.log/var/log/ceph/ceph-client.admin.52105.log: 2024-09-17T05:39:07.328 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41790.log 2024-09-17T05:39:07.328 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45962.log 2024-09-17T05:39:07.328 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2024-09-17T05:39:07.328 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.0.45962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47269.log 2024-09-17T05:39:07.329 INFO:teuthology.orchestra.run.smithi117.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45962.log.gz 2024-09-17T05:39:07.329 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52720.log 2024-09-17T05:39:07.329 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56190.log 2024-09-17T05:39:07.329 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45120.log 2024-09-17T05:39:07.329 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.56190.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56190.log.gz 2024-09-17T05:39:07.329 INFO:teuthology.orchestra.run.smithi117.stderr: -- /var/log/ceph/ceph-client.admin.43372.log 2024-09-17T05:39:07.330 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55291.log 2024-09-17T05:39:07.330 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54107.log 2024-09-17T05:39:07.330 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42166.log 2024-09-17T05:39:07.330 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45957.log 2024-09-17T05:39:07.330 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55291.log.gz 2024-09-17T05:39:07.330 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54107.log.gz 2024-09-17T05:39:07.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46096.log 2024-09-17T05:39:07.331 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47098.log 2024-09-17T05:39:07.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.0.45957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45957.log.gz 2024-09-17T05:39:07.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37167.log 2024-09-17T05:39:07.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2024-09-17T05:39:07.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43834.log 2024-09-17T05:39:07.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55543.log 2024-09-17T05:39:07.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46190.log 2024-09-17T05:39:07.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55669.log 2024-09-17T05:39:07.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46190.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55543.log.gz 2024-09-17T05:39:07.332 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52381.log 2024-09-17T05:39:07.333 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37042.log 2024-09-17T05:39:07.333 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52381.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55669.log.gz 2024-09-17T05:39:07.333 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-09-17T05:39:07.333 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-09-17T05:39:07.333 INFO:teuthology.orchestra.run.smithi117.stderr:gzip/var/log/ceph/ceph-client.admin.44430.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55835.log 2024-09-17T05:39:07.334 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42462.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30641.log 2024-09-17T05:39:07.334 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55835.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.51971.log.gz 2024-09-17T05:39:07.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55835.log.gz 2024-09-17T05:39:07.334 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52633.log.gz 2024-09-17T05:39:07.334 INFO:teuthology.orchestra.run.smithi117.stderr: -- /var/log/ceph/ceph-client.admin.54330.log 2024-09-17T05:39:07.334 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.tmp-client.admin.30641.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30641.log.gz 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53790.log 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5/var/log/ceph/ceph-client.admin.54330.log: --verbose -- /var/log/ceph/ceph-client.admin.53979.log 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2%/var/log/ceph/ceph-client.admin.53790.log: -- replaced with /var/log/ceph/ceph-client.admin.54330.log.gz 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46527.log 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53790.log.gz 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43624.log 2024-09-17T05:39:07.335 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53979.log.gz 2024-09-17T05:39:07.336 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44660.log 2024-09-17T05:39:07.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50853.log 2024-09-17T05:39:07.336 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43624.log: /var/log/ceph/ceph-client.admin.44660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51929.log 2024-09-17T05:39:07.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50853.log: -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-17T05:39:07.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51929.log: -- /var/log/ceph/ceph-client.admin.41994.log 2024-09-17T05:39:07.337 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2024-09-17T05:39:07.337 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2024-09-17T05:39:07.337 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55627.log 2024-09-17T05:39:07.337 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47455.log.gz 2024-09-17T05:39:07.337 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47571.log.gz 2024-09-17T05:39:07.337 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47413.log.gz 2024-09-17T05:39:07.338 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47371.log 2024-09-17T05:39:07.338 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53056.log 2024-09-17T05:39:07.338 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55627.log.gz 2024-09-17T05:39:07.338 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50685.log.gz 2024-09-17T05:39:07.338 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50727.log.gz 2024-09-17T05:39:07.338 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53308.log 2024-09-17T05:39:07.339 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54863.log 2024-09-17T05:39:07.339 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51700.log.gz 2024-09-17T05:39:07.339 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51761.log.gz 2024-09-17T05:39:07.339 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52276.log 2024-09-17T05:39:07.339 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53308.log: /var/log/ceph/ceph-client.admin.54863.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51803.log.gz 2024-09-17T05:39:07.339 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43582.log 2024-09-17T05:39:07.340 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51845.log.gz 2024-09-17T05:39:07.340 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54863.log.gz/var/log/ceph/ceph-client.admin.52276.log: 2024-09-17T05:39:07.340 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.52147.log.gz 2024-09-17T05:39:07.340 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52191.log.gz 2024-09-17T05:39:07.340 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.54905.log 2024-09-17T05:39:07.341 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2024-09-17T05:39:07.341 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42210.log 2024-09-17T05:39:07.341 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54905.log.gz 2024-09-17T05:39:07.341 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51071.log 2024-09-17T05:39:07.342 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.42210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37246.log 2024-09-17T05:39:07.342 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47311.log 2024-09-17T05:39:07.342 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.37246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46679.log 2024-09-17T05:39:07.342 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.47311.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-09-17T05:39:07.342 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43498.log 2024-09-17T05:39:07.343 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54022.log 2024-09-17T05:39:07.343 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37410.log 2024-09-17T05:39:07.343 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43666.log 2024-09-17T05:39:07.343 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1%/var/log/ceph/ceph-client.admin.37410.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.54022.log.gz 2024-09-17T05:39:07.343 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53140.log 2024-09-17T05:39:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.43666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50769.log 2024-09-17T05:39:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52549.log 2024-09-17T05:39:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.50769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52013.log 2024-09-17T05:39:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52549.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T05:39:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.52013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53579.log 2024-09-17T05:39:07.344 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46805.log 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: 76.9% -- replaced with /var/log/ceph/ceph-client.admin.36825.log.gz 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.36768.log.gz 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: 78.9% -- replaced with /var/log/ceph/ceph-client.admin.36853.log.gz 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: 79.7% -- replaced with /var/log/ceph/ceph-client.admin.36881.log.gz 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: 79.1% -- replaced with /var/log/ceph/ceph-client.admin.36741.log.gz 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46359.log 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37166.log.gz 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46805.log: 86.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.53579.log.gz -5 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44431.log 2024-09-17T05:39:07.345 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38068.log 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.37399.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.37234.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.37371.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37404.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37387.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44431.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.37378.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37364.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54284.log 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38068.log: /var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54599.log 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55501.log 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.54284.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47140.log.gz 2024-09-17T05:39:07.346 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47182.log.gz 87.4% 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47098.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47048.log.gz -- replaced with /var/log/ceph/ceph-client.admin.47311.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4%/var/log/ceph/ceph-client.admin.54599.log: -- replaced with /var/log/ceph/ceph-client.admin.47227.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47371.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47269.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45204.log 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54599.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46890.log 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.55501.log: /var/log/ceph/ceph-client.admin.45204.log: gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55501.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51617.log 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.50625.log.gz 2024-09-17T05:39:07.347 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.46890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53224.log 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.51617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45442.log 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.53224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44098.log 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.45442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56003.log 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.44098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54457.log 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.56003.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51071.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51175.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51428.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56003.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51217.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51115.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.54457.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52233.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51554.log.gz 2024-09-17T05:39:07.348 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.54457.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46443.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51344.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51259.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52423.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46527.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50853.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41790.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51512.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41994.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42210.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42166.log.gz 2024-09-17T05:39:07.349 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43295.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43372.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43414.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44431.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44775.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45120.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45204.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52339.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52507.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52678.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52720.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52762.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52804.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52846.log.gz 2024-09-17T05:39:07.350 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53098.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53182.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53308.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36964.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37012.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.37042.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37167.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.37246.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38164.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43540.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.37410.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43582.log.gz 2024-09-17T05:39:07.351 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43624.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43666.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43708.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43750.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43834.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44100.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44095.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44087.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.44098.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46679.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46721.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.52381.log.gz 2024-09-17T05:39:07.352 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52276.log.gz 2024-09-17T05:39:07.353 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53140.log.gz 2024-09-17T05:39:07.353 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53224.log.gz 2024-09-17T05:39:07.353 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45634.log.gz 2024-09-17T05:39:07.353 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46096.log.gz 2024-09-17T05:39:07.353 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T05:39:07.353 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46190.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46359.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50811.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46763.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46847.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46805.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.43010.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43498.log.gz 2024-09-17T05:39:07.357 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44660.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45442.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37827.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37867.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50769.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50937.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51617.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52013.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51929.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52105.log.gz 2024-09-17T05:39:07.358 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38068.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50987.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51029.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51386.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51887.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41593.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41675.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45289.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45331.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52549.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52591.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53014.log.gz 2024-09-17T05:39:07.359 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53056.log.gz 2024-09-17T05:39:07.392 INFO:teuthology.orchestra.run.smithi117.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-17T05:39:07.432 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46890.log.gz 2024-09-17T05:39:07.538 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-17T05:39:07.565 INFO:teuthology.orchestra.run.smithi117.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-17T05:39:07.650 INFO:teuthology.orchestra.run.smithi117.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T05:39:07.855 INFO:teuthology.orchestra.run.smithi119.stderr: 92.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T05:39:07.877 INFO:teuthology.orchestra.run.smithi119.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-17T05:39:07.913 INFO:teuthology.orchestra.run.smithi119.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T05:39:08.021 INFO:teuthology.orchestra.run.smithi119.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-17T05:39:08.241 INFO:teuthology.orchestra.run.smithi119.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T05:39:09.500 INFO:teuthology.orchestra.run.smithi119.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T05:39:10.142 INFO:teuthology.orchestra.run.smithi117.stderr: 95.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-09-17T05:39:11.474 INFO:teuthology.orchestra.run.smithi117.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T05:39:11.814 INFO:teuthology.orchestra.run.smithi119.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T05:45:33.662 INFO:teuthology.orchestra.run.smithi119.stderr: 95.3% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-09-17T05:58:01.096 INFO:teuthology.orchestra.run.smithi119.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T06:00:01.614 INFO:teuthology.orchestra.run.smithi119.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-17T06:00:30.126 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T06:00:49.144 INFO:teuthology.orchestra.run.smithi117.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T06:04:38.150 INFO:teuthology.orchestra.run.smithi117.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T06:04:41.027 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T06:04:41.030 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-17T06:04:41.030 INFO:teuthology.orchestra.run.smithi117.stderr:real 25m33.859s 2024-09-17T06:04:41.030 INFO:teuthology.orchestra.run.smithi117.stderr:user 8m37.061s 2024-09-17T06:04:41.030 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m23.870s 2024-09-17T06:04:41.801 INFO:teuthology.orchestra.run.smithi119.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T06:04:48.677 INFO:teuthology.orchestra.run.smithi119.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-17T06:04:48.679 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-09-17T06:04:48.679 INFO:teuthology.orchestra.run.smithi119.stderr:real 25m41.504s 2024-09-17T06:04:48.679 INFO:teuthology.orchestra.run.smithi119.stderr:user 11m6.845s 2024-09-17T06:04:48.679 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m30.042s 2024-09-17T06:04:48.679 INFO:tasks.ceph:Archiving logs... 2024-09-17T06:04:48.680 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/remote/smithi117/log 2024-09-17T06:04:48.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T06:06:32.577 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/remote/smithi119/log 2024-09-17T06:06:32.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T06:08:24.860 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T06:08:24.870 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T06:08:24.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T06:08:24.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T06:08:24.986 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T06:08:24.996 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T06:08:24.996 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T06:08:24.999 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T06:08:25.014 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-09-17T06:08:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-17T06:08:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T06:08:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-17T06:08:25.058 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 3 7 377 30 +699us[ +699us] +/- 48ms 2024-09-17T06:08:25.058 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -997us[ -997us] +/- 60ms 2024-09-17T06:08:25.058 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 95 -148us[ -151us] +/- 87ms 2024-09-17T06:08:25.058 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:08:25.061 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T06:08:25.062 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-09-17T06:08:25.062 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv01.front.sepia.ceph.com 3 7 377 97 +728us[ +676us] +/- 47ms 2024-09-17T06:08:25.062 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -1039us[-1039us] +/- 60ms 2024-09-17T06:08:25.062 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -176us[ -176us] +/- 88ms 2024-09-17T06:08:25.062 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T06:08:25.063 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T06:08:25.072 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T06:08:25.072 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T06:08:25.080 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T06:08:25.205 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T06:08:25.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T06:08:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:08:25.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T06:08:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:08:25.359 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T06:08:25.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T06:08:25.378 INFO:teuthology.task.internal:Duration was 2857.171245 seconds 2024-09-17T06:08:25.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T06:08:25.388 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T06:08:25.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T06:08:25.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T06:08:25.453 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T06:08:25.474 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T06:08:25.759 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T06:08:25.759 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-09-17T06:08:25.759 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T06:08:25.781 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-09-17T06:08:25.781 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T06:08:25.813 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T06:08:25.814 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T06:08:25.823 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T06:08:25.890 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T06:08:25.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T06:08:25.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T06:08:26.049 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T06:08:26.060 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T06:08:26.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T06:08:26.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T06:08:26.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T06:08:26.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T06:08:26.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T06:08:26.162 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-09-17T06:08:26.188 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-09-17T06:08:26.206 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T06:08:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:08:26.241 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T06:08:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:08:26.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T06:08:26.271 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T06:08:26.272 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/remote/smithi117 2024-09-17T06:08:26.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T06:08:26.319 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908618/remote/smithi119 2024-09-17T06:08:26.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T06:08:26.353 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T06:08:26.353 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T06:08:26.356 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T06:08:26.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T06:08:26.414 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T06:08:26.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T06:08:26.423 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T06:08:26.423 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T06:08:26.425 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T06:08:26.438 INFO:teuthology.orchestra.run.smithi117.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 06:08 /home/ubuntu/cephtest 2024-09-17T06:08:26.457 INFO:teuthology.orchestra.run.smithi119.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 06:08 /home/ubuntu/cephtest 2024-09-17T06:08:26.458 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T06:08:26.469 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T06:08:26.492 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} duration: 2857.1712448596954 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-17T06:08:26.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T06:08:26.582 INFO:teuthology.run:pass